SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\alpha_{t}^{\sigma} = \sqrt{\min\left(\sigma^{2},\bar{\nu}_{z,t - 1}\right)}\alpha_{t}$  and  $\alpha_{t} = \sqrt{d\log\left(\frac{1 + tL^{2} / \lambda}{\delta}\right)}$
Let action  $x_{t}$  be selected in the round  $t$ . Then, the instantaneous regret is given as follows:
$$
\begin{array}{l} r _ {t} = \max  _ {x \in \mathcal {X}} x ^ {\top} \theta^ {\star} - x _ {t} ^ {\top} \theta^ {\star} \\ = x ^ {\star} ^ {\top} \theta^ {\star} - x _ {t} ^ {\top} \theta^ {\star} \quad \left(\text {a s} x ^ {\star} = \max  _ {x \in \mathcal {X}} x ^ {\top} \theta^ {\star}\right) \\ = \left(x ^ {\star} - x _ {t}\right) ^ {\top} \theta^ {\star} \\ = \left(x ^ {\star} - x _ {t}\right) ^ {\top} \theta^ {\star} + \left(x ^ {\star} - x _ {t}\right) ^ {\top} \hat {\theta} _ {t} - \left(x ^ {\star} - x _ {t}\right) ^ {\top} \hat {\theta} _ {t} \\ = \left(x ^ {\star} - x _ {t}\right) ^ {\top} \hat {\theta} _ {t} - \left(x ^ {\star} - x _ {t}\right) ^ {\top} \left(\hat {\theta} _ {t} - \theta^ {\star}\right). \\ \end{array}
$$
A sub-optimal action is only selected when its upper confidence bound is larger than the optimal action. Then, if  $\left\| \hat{\theta}_t - \theta^\star \right\|_{\overline{V}_t} = \alpha_t^\sigma + \lambda^{1/2} S$ , then we have
$$
\begin{array}{l} r _ {t} \leq \alpha_ {t} \| x _ {t} \| _ {\bar {V} _ {t} ^ {- 1}} - \alpha_ {t} \| x ^ {\star} \| _ {\bar {V} _ {t} ^ {- 1}} - \left(x ^ {\star} - x _ {t}\right) ^ {\top} \left(\hat {\theta} _ {t} - \theta^ {\star}\right) \\ \leq \alpha_ {t} \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} - \alpha_ {t} \| x ^ {\star} \| _ {\overline {{V}} _ {t} ^ {- 1}} - \| x ^ {\star} - x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} \| \hat {\theta} _ {t} - \theta^ {\star} \| _ {\overline {{V}} _ {t}} \\ \leq \alpha_ {t} \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} - \alpha_ {t} \| x ^ {\star} \| _ {\overline {{V}} _ {t} ^ {- 1}} + \alpha_ {t} \| x ^ {\star} - x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} \\ = \alpha_ {t} \left(\| x _ {t} \| _ {\bar {V} _ {t} ^ {- 1}} - \| x ^ {\star} \| _ {\bar {V} _ {t} ^ {- 1}} + \| x ^ {\star} - x _ {t} \| _ {\bar {V} _ {t} ^ {- 1}}\right) \\ \leq \alpha_ {t} \left(\| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} - \| x ^ {\star} \| _ {\overline {{V}} _ {t} ^ {- 1}} + \left\| X _ {t, a _ {t} ^ {\star}} \right\| _ {\overline {{V}} _ {t} ^ {- 1}} + \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}}\right) \\ = 2 \alpha_ {t} \| x _ {t} \| _ {V _ {t} ^ {- 1}} \\ \Longrightarrow r _ {t} \leq 2 \left(\alpha_ {t} ^ {\sigma} + \lambda^ {1 / 2} S\right) \| x _ {t} \| _ {\bar {V} _ {t} ^ {- 1}}. \\ \end{array}
$$
Let  $X_{t} = \{x_{s}\}_{s = 1}^{t}$ . For  $t > q + 2$  and  $\bar{\nu}_{z,t} < \sigma^2$ , the expected instantaneous regret of OFUL-AF is
$$
\begin{array}{l} \mathbb {E} \left[ r _ {t} (\text {O F U L - A F}) \right] \leq \mathbb {E} \left[ 2 \left(\alpha_ {t} ^ {\sigma} + \lambda^ {1 / 2} S\right) \| x _ {t} \| _ {\bar {V} _ {t} ^ {- 1}} \right] \\ = \mathbb {E} \left[ 2 \left(\sqrt {\bar {\nu} _ {z , t}} \alpha_ {t} + \lambda^ {1 / 2} S\right) \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} \right] \\ = 2 \mathbb {E} \left[ \mathbb {E} \left[ \left(\sqrt {\bar {\nu} _ {z , t}} \alpha_ {t} + \lambda^ {1 / 2} S\right) \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} | \boldsymbol {X} _ {t} \right] \right] \\ = 2 \mathbb {E} \left[ \alpha_ {t} \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} \mathbb {E} \left[ \sqrt {\bar {\nu} _ {z , t}} | \boldsymbol {X} _ {t} \right] + \lambda^ {1 / 2} S \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} \right] \\ \leq 2 \alpha_ {t} \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} \mathbb {E} \left[ \mathbb {E} \left[ \sqrt {\frac {(t - 2) \hat {\nu} _ {z , t - 1}}{\chi_ {1 - \delta , t} ^ {2}}} | \boldsymbol {X} _ {t} \right] \right] + 2 \lambda^ {1 / 2} S \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} \\ = 2 \alpha_ {t} \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} \sqrt {\frac {(t - 2)}{\chi_ {1 - \delta , t} ^ {2}}} \mathbb {E} \left[ \sqrt {\hat {\nu} _ {z , t - 1}} \right] + 2 \lambda^ {1 / 2} S \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}}. \\ \end{array}
$$
Since  $\hat{\nu}_{z,t-1}$  is an unbiased estimator of the sample variance of hybrid rewards,  $\mathbb{E}[\hat{\nu}_{z,t-1}] = \mathbb{V}(z_{s,q})$  for  $s \in \{1,\dots,t\}$ . Using Theorem 1, we have  $\mathbb{V}(z_{s,q}) = \left(1 + \frac{q}{t - q - 3}\right)(1 - \rho^2)\sigma^2 =$
$\left(\frac{(t - 3)(1 - \rho^2)}{t - q - 3}\right)\sigma^2$  as  $t^{\mathrm{th}}$  observation is not available at the beginning of the round  $t$ . With increasing  $t$ ,  $C_t = \sqrt{\frac{(t - 2)}{\chi_{1 - \delta, t}^2}}$  tends to 1. With all these observations, we have
$$
\begin{array}{l} \mathbb {E} \left[ r _ {t} (\text {O F U L - A F}) \right] \leq 2 C _ {t} \left(\frac {(t - 3) (1 - \rho^ {2})}{t - q - 3}\right) ^ {\frac {1}{2}} \sigma \alpha_ {t} \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} + 2 \lambda^ {1 / 2} S \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} \\ = 2 \left(C _ {t} \left(\frac {(t - 3) (1 - \rho^ {2})}{t - q - 3}\right) ^ {\frac {1}{2}} \sigma \alpha_ {t} + \lambda^ {1 / 2} S\right) \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} \\ \end{array}
$$
Let  $r_t(\mathrm{OFUL})$  be the upper bound on instantaneous regret for OFUL algorithm, i.e.,  $r_t(\mathrm{OFUL}) = 2\left(\sigma \alpha_t + \lambda^{1/2} S\right) \| x_t \|_{\overline{V}_t^{-1}}$ . Then, we have
$$
\begin{array}{l} \mathbb {E} \left[ r _ {t} (\text {O F U L - A F}) \right] \leq 2 C _ {t} \left(\frac {(t - 3) (1 - \rho^ {2})}{t - q - 3}\right) ^ {\frac {1}{2}} \left(\sigma \alpha_ {t} + \lambda^ {1 / 2} S\right) \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} \\ + 2 \left(1 - C _ {t} \left(\frac {(t - 3) (1 - \rho^ {2})}{t - q - 3}\right) ^ {\frac {1}{2}}\right) \lambda^ {1 / 2} S \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}} \\ \leq C _ {t} \left(\frac {(t - 3) (1 - \rho^ {2})}{t - q - 3}\right) ^ {\frac {1}{2}} r _ {t} (\text {O F U L}) \\ + 2 \left(1 - C _ {t} \left(\frac {(t - 3) (1 - \rho^ {2})}{t - q - 3}\right) ^ {\frac {1}{2}}\right) \lambda^ {1 / 2} S \| x _ {t} \| _ {\overline {{V}} _ {t} ^ {- 1}}. \\ \Rightarrow \mathbb {E} \left[ r _ {t} (\text {O F U L - A F}) \right] \leq \widetilde {O} \left(\left(\frac {(t - 3) (1 - \rho^ {2})}{t - q - 3}\right) ^ {\frac {1}{2}} r _ {t} (\text {O F U L})\right). \\ \end{array}
$$
$$
r _ {t} (\mathfrak {A} - \mathrm {A F}) \leq 2 \min  (\sigma , (\bar {\nu} _ {e, z, t}) ^ {\frac {1}{2}}) h (x, \mathcal {O} _ {t}) + l (x, \mathcal {O} _ {t}),
$$
where  $e = \{IS, MF, KF\}$ , and  $KF$  denotes the case where auxiliary functions are known. For  $t > q + 2$  and  $\bar{\nu}_{e,z,t} < \sigma^2$ ,  $\mathbb{E}\left[r_t(\mathfrak{A}-\mathrm{AF})\right] \leq \widetilde{O}\left(\left(\left(\frac{t-(1-a(e))q-3}{t-q-3}\right)(1-\rho_e^2)\right)^{\frac{1}{2}}r_t(\mathfrak{A})\right)$ , where  $a(KF) = 1$ .
Proof. Let  $\mathfrak{A}$  be an AFC bandit algorithm with  $|f_t^{\mathfrak{A}}(x) - f(x)| \leq \sigma h(x, \mathcal{O}_t) + l(x, \mathcal{O}_t)$  and  $\bar{\nu}_{e,z,t}$  be the upper bound on sample variance of hybrid reward. After  $\mathfrak{A}$  uses hybrid rewards for estimating function  $f$ , then, with probability at least  $1 - 2\delta$
$$
\left| f _ {t} ^ {\mathfrak {A}} (x) - f (x) \right| \leq \min  \left(\sigma , \left(\bar {\nu} _ {e, z, t}\right) ^ {\frac {1}{2}}\right) h (x, \mathcal {O} _ {t}) + l (x, \mathcal {O} _ {t}) \tag {8}
$$
The proof follows similar steps as the first part of the proof of Theorem 2. The only key difference is the upper bound of variance of hybrid rewards, which depends on the underlying sampling strategy based on whether auxiliary functions are known or unknown. The upper bound on sample variance is given by  $\bar{\nu}_{e,z,t} = \frac{(t - 2)\hat{\nu}_{e,z,t - 1}}{\chi_{1 - \delta,t}^2}$ , where  $\hat{\nu}_{e,z,t - 1}$  is an unbiased sample variance estimate of hybrid rewards using  $t - 1$  observations with sampling strategy  $e$  and  $\chi_{1 - \delta ,t}^{2}$  (implying the variance upper bound holds with at least probability of  $1 - \delta$ ) denotes  $100(1 - \delta)^{\mathrm{th}}$  percentile value of the chi-squared distribution with  $t - 2$  degrees of freedom.
$$
r _ {t} = \max  _ {x \in \mathcal {X}} f (x) - f (x _ {t}) = f \left(x ^ {\star}\right) - f (x _ {t}) \quad \left(\text {a s} x ^ {\star} = \max  _ {x \in \mathcal {X}} f (x)\right)
$$
$$
\begin{array}{l} \leq \left| f _ {t} ^ {\mathfrak {A}} \left(x ^ {\star}\right) + \min  \left(\sigma , \left(\bar {\nu} _ {e, z, t}\right) ^ {\frac {1}{2}}\right) h \left(x ^ {\star}, \mathcal {O} _ {t}\right) + l \left(x ^ {\star}, \mathcal {O} _ {t}\right) - f \left(x _ {t}\right) \right| \\ \leq \left| f _ {t} ^ {\mathfrak {A}} (x _ {t}) + \min  (\sigma , (\bar {\nu} _ {e, z, t}) ^ {\frac {1}{2}}) h (x _ {t}, \mathcal {O} _ {t}) + l (x _ {t}, \mathcal {O} _ {t}) - f (x _ {t}) \right| \\ \leq \left| f _ {t} ^ {\mathfrak {A}} \left(x _ {t}\right) - f (x _ {t}) \right| + \min  \left(\sigma , \left(\bar {\nu} _ {e, z, t}\right) ^ {\frac {1}{2}}\right) h \left(x _ {t}, \mathcal {O} _ {t}\right) + l \left(x _ {t}, \mathcal {O} _ {t}\right) \\ \leq 2 \min  (\sigma , (\bar {\nu} _ {e, z, t}) ^ {\frac {1}{2}}) h (x _ {t}, \mathcal {O} _ {t}) + l (x _ {t}, \mathcal {O} _ {t}), \\ \end{array}
$$
in which the first and last inequalities have used the upper bound given in Eq. (8), and the second inequality follows because actions are selected using the upper confidence bounds. The remaining proof will follow the similar steps as the second part of Theorem 2 except using Theorem 3 instead of Theorem 1 for quantifying the variance reduction due to hybrid rewards when IS or MF sampling strategy is used for estimating auxiliary feedback function.
A.4 Auxiliary feedback in contextual bandits
Many real-life applications have some additional information readily available for the learner before selecting an action, e.g., users' profile information is known to the online platform before making any recommendations. Such information is treated as contextual information in bandit literature, and the bandit problem having contextual information is refereed as contextual bandits (Li et al., 2010). Since the value of the reward function also depends on the context, the learner's goal is to use contextual information to select a better action.
We extend our results for the contextual bandits problem. In this setting, we assume that a learner has been given an action set denoted by  $\mathcal{A}$ . In round  $t$ , the environment generates a vector  $(x_{t,a}, y_{t,a}, \{w_{t,a,i}\}_{i=1}^{q})$  for each action  $a \in \mathcal{A}$ . Here,  $x_{t,a}$  is the context-action  $d$ -dimensional feature vector of observed context in round  $t$  and action  $a$ ,  $y_{t,a}$  is the stochastic reward received for context-action pair  $x_{t,a}$ , and  $w_{t,a,i}$  is the  $i^{\text{th}}$  auxiliary feedback associated with the reward  $y_{t,a}$ . We assume that the reward is a function of the context-action pair  $x_{t,a}$ , which is given as  $y_{t,a} = f(x_{t,a}) + \varepsilon_t$ , where  $f: \mathbb{R}^d \to \mathbb{R}$  is an unknown function and  $\varepsilon_t$  is a zero-mean Gaussian noise with variance  $\sigma^2$ . The auxiliary feedback is also assumed to be a function of the context-action pair  $x_{t,a}$ , given as  $W_{t,a,i} = g_i(x_{t,a}) + \varepsilon_{t,i}^w$ , where  $g_i: \mathbb{R}^d \to \mathbb{R}$  and  $\varepsilon_{t,i}^w$  is a zero-mean Gaussian noise with variance  $\sigma_w^2$ . The correlation coefficient between reward and associated auxiliary feedback is denoted by  $\rho$ .
We denote the optimal action for a context observed in the round  $t$  as  $a_{t}^{\star} = \operatorname{argmax}_{a \in \mathcal{A}} f(x_{t,a})$ . The interaction between a learner and its environment is given as follows. At the beginning of round  $t$ , the environment generates a context, and then the learner selects an action  $a_{t}$  from action set  $\mathcal{A}$  for that context using past information of context-actions feature vector, observed rewards and its associated auxiliary feedback until round  $t - 1$ . After selecting action  $a_{t}$ , the learner receives a reward  $(y_{t,a_{t}})$  with its associated auxiliary feedback and incurs a penalty (or instantaneous regret)  $r_{t}$  where  $r_{t} = f(x_{t,a_{t}^{\star}}) - f(x_{t,a_{t}})$ . We aim to learn a sequential policy that selects actions to minimize the total penalty and evaluate the performance of such policy through regret, which is the sum of the penalty incurred by the learner. Formally, for  $T$  contexts, the regret of a policy  $\pi$  that selects action  $a_{t}$  for a context observed in round  $t$  is given by
$$
\Re_ {T} (\pi) = \sum_ {t = 1} ^ {T} \left(f \left(x _ {t, a _ {t} ^ {*}}\right) - f \left(x _ {t, a _ {t}}\right)\right). \tag {9}
$$
A policy  $\pi$  is a good policy when it has sub-linear regret. This implies that the policy will eventually learn to recommend the best action for every context. Similar to the parameterized bandit problem case, we can use the existing contextual bandit algorithms, which are AFC bandit algorithms. Depending on the problem, an appropriate AFC contextual bandit algorithm is selected that uses hybrid rewards to estimate reward function. The smaller variance of hybrid rewards leads to tighter upper confidence bound of the unknown reward function and hence smaller regret.
A.5 More details about experiments
To demonstrate the performance gain from using auxiliary feedback, we have considered three different bandit settings: linear bandits, linear contextual bandits, and non-linear contextual bandits. The details of the problem instance used in our experiments are as follows.
(a) Biased Estimator
(b) Estimator from historical data
(c) Influence of correlation
(f) Influence of correlation
Figure 3: Top row: Experiment using linear bandits problem instance. Bottom row: Experiment using non-linear contextual bandits problem instance. Left to right:: Regret vs. different biases (left figure), regret vs. number of historical samples of auxiliary feedback (middle figure), and regret vs. varying correlation coefficients of reward and its auxiliary feedback (right figure).
Linear bandits: We use a 5-dimensional space in which each sample is represented by  $x = (x_{1},\ldots ,x_{5})$ , where the value of  $x_{j}$  is restricted in  $(-3,3)$ . We randomly select a 5-dimensional vector  $\theta^{\star}$  with a unit norm whose each value is restricted in  $(0,1)$ . In all linear bandits experiments, we use  $\lambda = 0.01$ ,  $L = 2.236$ ,  $S = 1$ , and  $\delta = 0.05$ . In round  $t$ , the reward for selected action  $x_{t}$  is
$$
y _ {t} = v _ {t} + w _ {t},
$$
where  $v_{t} = x_{t}^{\top}\theta_{v}^{\star} + \varepsilon_{t}^{v}$  and  $w_{t} = x_{t}^{\top}\theta_{w}^{\star} + \varepsilon_{t}^{w}$ . We set  $\theta_v^\star = (0,\theta_2^\star ,0,\theta_4^\star ,0)$  and  $\theta_w^\star = (\theta_1^\star ,0,\theta_3^\star ,0,\theta_5^\star)$ . As we treat  $w_{t}$  as auxiliary feedback,  $\theta_w^\star$  may be assumed to be known in some experiments. The random noise  $\varepsilon_t^v$  is zero-mean Gaussian noise with variance  $\sigma_v^2$ . Whereas  $\varepsilon_t^w$  is also zero-mean Gaussian noise, but the variance is  $\sigma_w^2$ . We assumed that  $\sigma^2 = \sigma_v^2 +\sigma_w^2$  is known, but not the  $\sigma_v^2$  and  $\sigma_w^2$ . The default value of  $\sigma_v^2 = 0.01$  and  $\sigma_w^2 = 0.01$ . It can be easily shown that the correlation coefficient of  $y_{t}$  and  $w_{t}$  is  $\rho = \sqrt{\sigma_w^2 / (\sigma_v^2 + \sigma_w^2)}$ . We run each experiment for 5000 rounds.
Linear contextual bandits: We first generate a 2-dimensional synthetic dataset with 5000 data samples. Each sample is represented by  $x = (x_{1},x_{2})$ , where the value of  $x_{j}$  is drawn uniformly at random from  $(-1,1)$ . Our action set  $\mathcal{A}$  has four actions:  $\{(x_1,x_2),(x_1, - x_2),(-x_1,x_2),(-x_1, - x_2)\}$ . We uniformly generate a  $\theta^{\star}$  such that its norm is 1. In all experiments, the data samples are treated as contexts, and we use  $\lambda = 0.01$ ,  $L = 1.41$ ,  $S = 1$ , and  $\delta = 0.05$ . The observed reward for a context-action feature vector has two components. We treated one of the components as auxiliary feedback. In round  $t$ , the reward context-action feature vector  $x_{t,a}$  is given as follows:
$$
y _ {t, a _ {t}} = v _ {t, a _ {t}} + w _ {t, a _ {t}},
$$
where  $v_{t,a_t} = x_{t,a}^\top \theta_v^\star + \varepsilon_t^v$  and  $w_{t,a_t} = x_{t,a}^\top \theta_w^\star + \varepsilon_t^w$ . We set  $\theta_v^\star = (0, \theta_2^\star, 0, \theta_4^\star)$  and  $\theta_w^\star = (\theta_1^\star, 0, \theta_3^\star, 0)$ . As we treat  $w_{t,a_t}$  as auxiliary feedback,  $\theta_w^\star$  is known for some experiments. The random noise  $\varepsilon_t^v$  is zero-mean Gaussian noise with variance  $\sigma_v^2$ . Whereas  $\varepsilon_t^w$  is also zero-mean Gaussian noise, but the variance is  $\sigma_w^2$ . We assumed that  $\sigma^2 = \sigma_v^2 + \sigma_w^2$  is known, but not the  $\sigma_v^2$  and  $\sigma_w^2$ . The default value of  $\sigma_v^2 = 0.01$  and  $\sigma_w^2 = 0.01$ . It can be easily shown that the correlation coefficient of  $y_{t,a}$  and  $w_{t,a}$  is  $\rho = \sqrt{\sigma_w^2 / (\sigma_v^2 + \sigma_w^2)}$ .
(b) Linear Contextual Bandits
Figure 4: Comparing regret vs. the number of historical samples of auxiliary feedback in different settings. In this experiment, historical samples for each run are randomly generated as compared to Fig. 2e, Fig. 3b, and Fig. 3e where history is kept fixed across the runs.
Non-linear contextual bandits: This problem instance is adapted from the linear contextual bandits problem instance. We first generate a 2-dimensional synthetic dataset with 5000 data samples. Each sample is represented by  $x = (x_{1},x_{2})$ , where the value of  $x_{j}$  is drawn uniformly at random from  $(-1,1)$ . We then use a polynomial kernel with degree 2 to have a non-linear transformation of samples. We removed (i.e., bias) the first (i.e., 1) and last value  $(i.e.,x_2^2)$  from the transformed samples, which reduced the dimensional of each transformed sample to 4 and represented as  $(x_{1},x_{2},x_{1}^{2},x_{1}x_{2})$ , which is used as context. For this setting, the action set  $\mathcal{A}$  has six actions:  $\{(x_{1},x_{2}, - x_{1}^{2}, - x_{1}x_{2}),(x_{1}, - x_{2},x_{1}^{2}, - x_{1}x_{2}),(-x_{1},x_{2},x_{1}^{2}, - x_{1}x_{2}),(x_{1}, - x_{2}, - x_{1}^{2},x_{1}x_{2}),$ $(-x_{1},x_{2}, - x_{1}^{2},x_{1}x_{2}),(-x_{1}, - x_{2},x_{1}^{2},x_{1}x_{2}),\}$ . We uniformly generate a  $\theta^{\star}$  such that its norm is 1. In all experiments, we use  $\lambda = 0.01$ ,  $L = 2$ ,  $S = 1$ , and  $\delta = 0.05$ . The observed reward for a context-action feature vector has two components. We treated one of the components as auxiliary feedback. In round  $t$ , the reward context-action feature vector  $x_{t,a}$  is given as follows:
where  $v_{t,a_t} = x_{t,a}^\top \theta_v^\star + \varepsilon_t^v$  and  $w_{t,a_t} = x_{t,a}^\top \theta_w^\star + \varepsilon_t^w$ . We set  $\theta_v^\star = (0, \theta_2^\star, 0, \theta_4^\star, 0, \theta_6^\star, 0, \theta_8^\star)$  and  $\theta_w^\star = (\theta_1^\star, 0, \theta_3^\star, 0, \theta_5^\star, 0, \theta_7^\star, 0)$ . As we treat  $w_{t,a_t}$  as auxiliary feedback,  $\theta_w^\star$  is known for some experiments. The random noise  $\varepsilon_t^v$  is zero-mean Gaussian noise with variance  $\sigma_v^2$ . Whereas  $\varepsilon_t^w$  is also zero-mean Gaussian noise, but the variance is  $\sigma_w^2$ . We assumed that  $\sigma^2 = \sigma_v^2 + \sigma_w^2$  is known, but not the  $\sigma_v^2$  and  $\sigma_w^2$ . The default value of  $\sigma_v^2 = 0.01$  and  $\sigma_w^2 = 0.01$ . It can be easily shown that the correlation coefficient of  $y_{t,a}$  and  $w_{t,a}$  is  $\rho = \sqrt{\sigma_w^2 / (\sigma_v^2 + \sigma_w^2)}$ .
Regret with varying correlation coefficient: As the correlation coefficient of reward and auxiliary feedback is  $\rho = \sqrt{\sigma_w^2 / (\sigma_v^2 + \sigma_w^2)}$ , we varied  $\sigma_v$  over the values  $\{0.3, 0.2, 0.1528, 0.1, 0.0655\}$  to obtain problem instances with different correlation coefficient for all problem instances.
Variance estimation: Since the value of  $\sigma^2$  is known in all our experiments, we directly estimate the correlation coefficient  $(\rho)$  as  $\hat{\rho} = \operatorname{Cov}(y, w) / (\sqrt{\mathbb{V}(w)} \sigma)$ . Then, use it to set  $\bar{\nu}_{e,z,t} = (1 - \hat{\rho}^2) \sigma^2$ .