Title: Exploiting Correlated Auxiliary Feedback in Parameterized Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: Parameterized Bandits

Auxiliary Feedback

Control Variate

Regret Minimization
Abstract: We study a novel variant of the parameterized bandits problem in which the learner can observe additional auxiliary feedback that is correlated with the observed reward. The auxiliary feedback is readily available in many real-life applications, e.g., an online platform that wants to recommend the best-rated services to its users can observe the user's rating of service (rewards) and collect additional information like service delivery time (auxiliary feedback). In this paper, we first develop a method that exploits auxiliary feedback to build a reward estimator with tight confidence bounds, leading to a smaller regret. We then characterize the regret reduction in terms of the correlation coefficient between reward and its auxiliary feedback. Experimental results in different settings also verify the performance gain achieved by our proposed method.

Exploiting Correlated Auxiliary Feedback in Parameterized Bandits

Abstract

We study a novel variant of the parameterized bandits problem in which the learner can observe additional auxiliary feedback that is correlated with the observed reward. The auxiliary feedback is readily available in many real-life applications, e.g., an online platform that wants to recommend the best-rated services to its users can observe the user's rating of service (rewards) and collect additional information like service delivery time (auxiliary feedback). In this paper, we first develop a method that exploits auxiliary feedback to build a reward estimator with tight confidence bounds, leading to a smaller regret. We then characterize the regret reduction in terms of the correlation coefficient between reward and its auxiliary feedback. Experimental results in different settings also verify the performance gain achieved by our proposed method.

1 Introduction

Parameterized bandits (Slivkins, 2019; Lattimore and Szepesvári, 2020) have many real-life applications in online recommendation, advertising, web search, and e-commerce. In this bandit problem, a learner selects an action and receives a reward for the selected action. Due to the large (or infinite) number of actions, the mean reward of each action is assumed to be parameterized by an unknown function, e.g., linear (Li et al., 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011; Agrawal and Goyal, 2013), GLM (Filippi et al., 2010; Li et al., 2017; Jun et al., 2017), and non-linear (Valko et al., 2013; Chowdhury and Gopalan, 2017). The learner aims to learn the best action as quickly as possible. However, it depends on the tightness of confidence bounds of function that correlate action with the reward. The learner exploits any available information like side information (i.e., information available to the learner before selecting an action, e.g., contexts) (Li et al., 2010; Agrawal and Goyal, 2013; Li et al., 2017) and side observations (i.e., information about other actions, e.g., graph feedback) (Alon et al., 2015; Wu et al., 2015) to make confidence bounds as tight as possible. This paper considers another type of additional information (correlated with the reward) that a learner can observe with reward for the selected action, which we call auxiliary feedback .

The auxiliary feedback is readily available in many real-life applications. For example, consider an online food delivery platform that wishes to recommend the best restaurants (actions) to its users. After receiving food, the platform observes user ratings (rewards) for the order and can collect additional information like food delivery time (auxiliary feedback). Since the restaurant's rating also depends on overall food delivery time, one can expect it to be correlated with the user rating. The platform can estimate or know the average delivery time for a given order from historical data. Similar scenarios arise in recommending the best cab to users (auxiliary information can be the cab's distance from the rider or driver's response to ride request), e-commerce platforms choosing top sellers to buyers (auxiliary information can be seller's response time for order confirmation and delivery), queuing network (Lavenberg and Welch, 1981; Lavenberg et al., 1982), jobs scheduler (Verma and Hanawal, 2021), and many more. Therefore, the following question naturally arises: How to exploit correlated auxiliary feedback to improve the performance of a bandit algorithm?

One possible method is to use auxiliary feedback in the form of control variates (Lavenberg et al., 1982; Nelson, 1990) for the observed reward. A control variate represents any random variable (auxiliary feedback) with a known mean that is correlated with the random variable of interest (reward). Several works (Kreutzer et al., 2017; Sutton and Barto, 2018; Vlassis et al., 2021; Verma and Hanawal, 2021) have used control variates to estimate the mean reward estimator with smaller variance, leading to tight confidence bounds and hence better performance. The closest work to our setting is Verma and Hanawal (2021). However, it only focuses on the non-parameterized setting and assumes a finite number of actions. We thus consider a more general bandit setting with a large (or even infinite) number of actions and allow an unknown function to parameterize auxiliary feedback.

Motivated by control variate theory (Nelson, 1990), we first introduce hybrid reward , which combines the reward and its auxiliary feedback in such a way that hybrid reward is an unbiased reward estimator with smaller variance than the observed reward. However, the optimal combination of reward and its auxiliary feedback requires knowledge of the covariance matrix among auxiliary feedback and covariance between reward and its auxiliary feedback, which may not be available in practice. Since the reward and its auxiliary feedback are functions of the selected action, existing control variate results can not be useful to our sequential setting. Naturally, we face the question of how to combine reward and its auxiliary feedback efficiently using available information. To answer this, we extend control variate theory results to the problems where known functions can parameterize control variates (in Section 3) and then extend to setting where unknown functions parameterize control variates (in Section 4). These contributions are themselves of independent interest in control variate theory.

Equipped with these results, we show that the variance of hybrid rewards is smaller than observed rewards (Theorem 1 and Theorem 3). We then propose a method that uses hybrid rewards instead of observed rewards for estimating reward function, resulting in tight confidence bounds and, hence, lower regret. We introduce the notion of Auxiliary Feedback Compatible (AFC) bandit algorithm. An AFC bandit algorithm can use hybrid rewards instead of only observed rewards. We prove that the expected instantaneous regret of any AFC bandit algorithm using hybrid rewards is smaller by a factor of O ((1 -ρ 2 ) 1 2 ) compared to the same AFC bandit algorithm using only observed rewards, where ρ is the correlation coefficient of the reward and its auxiliary feedback (Theorem 2 and Theorem 4). Our experimental results in different settings also verify our theoretical results (in Section 5).

1.1 Related work

Several prior works use additional information to improve the performance of bandit algorithms. In the following, we discuss how auxiliary feedback differs from side information and side observation.

Side Information: Several works use context as side information to select the best action to play. This line of work is popularly known as contextual bandits (Li et al., 2010; Chu et al., 2011; Agrawal and Goyal, 2013; Li et al., 2017). Here, the mean reward of each arm is a function of context and is often parameterized, e.g., linear (Li et al., 2010; Chu et al., 2011; Agrawal and Goyal, 2013), GLM (Li et al., 2017), and non-linear (Valko et al., 2013). These contexts are assumed to be observed before an action is taken. However, we consider a problem where additional information is correlated with rewards that can only be observed after selecting the action.

Side Observations: Several works consider the different side observations settings in the literature, e.g., stochastic (Caron et al., 2012), adversarial (Mannor and Shamir, 2011; Kocák et al., 2014), graph feedback (Alon et al., 2015; Wu et al., 2015; Alon et al., 2017), and cascading feedback (Verma et al., 2019, 2020a,b). Side observations represent the additional information available about actions that the learner does not select . Auxiliary feedback is different from side information as it is available only for selected action and provides more information about the reward of that action.

Auxiliary Feedback: We use auxiliary feedback as control variates, which are used extensively for variance reduction in Monte-Carlo simulation of complex systems (Lavenberg and Welch, 1981; Lavenberg et al., 1982; James, 1985; Nelson, 1989, 1990; Botev and Ridder, 2017; Chen and Ghahramani, 2016). Recent works (Kreutzer et al., 2017; Vlassis et al., 2021; Verma and Hanawal, 2021) and (Sutton and Barto, 2018, Chapter 7.4) have exploited the availability of these control variates to build estimators with smaller variance and develop algorithms that have better performance guarantees. The closest work to our setting is Verma and Hanawal (2021). However, they only consider a non-parameterized setting with a finite number of actions. We thus consider a more general bandit setting with large (infinite) actions and allow a function to parameterize auxiliary feedback.

2 Problem setting

We consider a novel parameterized bandits problem in which the learner can observe auxiliary feedback correlated with the observed reward. In this problem, a learner has been given an action set, denoted by X ⊂ R d where d ≥ 1 . At the beginning of round t , the learner selects an action x t from action set X . Then, the environment generates a stochastic reward y t . = f ( x t ) + ε t for the selected action x t , where f : R d → R is an unknown reward function and ε t is a zero-mean Gaussian noise with variance σ 2 . Apart from the stochastic reward y t , the environment generates q of auxiliary feedback. The i -th auxiliary feedback is denoted by w t,i . = g i ( x t ) + ε w t,i , where g i : R d → R and ε w t,i is a zero-mean Gaussian noise with variance σ 2 w,i . The multiple correlation coefficient of reward and its auxiliary feedback is denoted by ρ and assumed to be the same across all actions.

The optimal action (denoted by x ⋆ ) has the maximum function value, i.e., x ⋆ ∈ argmax x ∈X f ( x ) . After selecting an action x t , the learner incurs a penalty (or instantaneous regret ) r t , where r t . = f ( x ⋆ ) -f ( x t ) . Since the optimal action is unknown, we sequentially estimate the reward function using available information on rewards and associated auxiliary feedback for the selected actions and then use it for choosing the action in the following round. Our goal is to learn a sequential policy that selects actions such that the total penalty (or regret ) incurred by the learner is as minimum as possible. After T rounds, the regret of a sequential policy π that selects action x t in the round t is given by

A policy π is a good policy if it has sub-linear regret, i.e., lim T →∞ R T ( π ) /T = 0 . This implies that the policy π will eventually learn to recommend the best action.

3 Known auxiliary feedback functions

We first focus on a simple case where all auxiliary feedback functions are assumed to be known. This assumption is not very strict in many applications as the learner can construct auxiliary feedback such that its mean value is known beforehand (see Kreutzer et al. (2017), Vlassis et al. (2021), and Chapter 12.9 of Sutton and Barto (2018) for such examples). When auxiliary feedback functions are unknown, we can estimate them using historical data or additional samples of auxiliary feedback. However, it will have some penalty in the performance (more details are in Section 4 and Section 5).

The first challenge we face is how to exploit auxiliary feedback to get a better reward function estimator . To resolve this, we extend control variate theory (Nelson, 1990) to the problems where a function can parameterize control variates. This new contribution is itself of independent interest.

3.1 Control variate

Let µ be the unknown variable that needs to be estimated and y be its unbiased estimator, i.e., E [ y ] = µ . Any random variable w with a known mean value ( ω ) can be treated as a control variate for y if it is correlated with y . The control variate method (Nelson, 1990) exploits errors in estimates of known random variables to reduce the estimator's variance for the unknown random variable. This method works as follows. For any choice of a coefficient β , define a new random variable as z . = y -β ( w -ω ) . Note that z is also an unbiased estimator of µ (i.e., E [ z ] = µ ) as

Using properties of variance and covariance, the variance of z is given by

The variance of z is minimized by setting β to β ⋆ = Cov ( y, w ) / V ( w ) and the minimum value is (1 -ρ 2 ) V ( y ) , where ρ = Cov ( y, w ) / √ V ( w ) V ( y ) is the correlation coefficient of y and w . We exploit this variance reduction to design a reward function estimator with tight confidence bounds.

3.2 Auxiliary feedback as control variates

Since the auxiliary feedback functions are known, we define a new variable using the reward sample and its auxiliary feedback. We refer to this variable as 'hybrid reward. ' The hybrid reward definition

is motivated by the control variate method, except the control variate is parameterized by function in our setting. As w s,i is the i th auxiliary feedback observed with reward y s , the hybrid reward for reward ( y s ) with its q auxiliary feedback { w s,i } q i =1 is defined by

where w s = ( w s, 1 , . . . , w s,q ) , g s = ( g 1 ( x s ) , . . . , g q ( x s )) , and β = ( β 1 , . . . , β q ) ⊤ . Let Σ ww ∈ R q × q be the covariance matrix among auxiliary feedback and σ yw ∈ R q × 1 be the vector of covariance between the reward and each of its auxiliary feedback. Then, the variance of z s,q is minimized by setting the coefficient vector β to β ⋆ = Σ -1 ww σ yw , and the minimum value is (1 -ρ 2 ) σ 2 , where ρ 2 = σ ⊤ yw Σ -1 ww σ yw /σ 2 is the multiple correlation coefficient of reward and its auxiliary feedback.

However, Σ ww and σ yw can be unknown in practice and need to be estimated to get the best estimate for β ⋆ to achieve maximum variance reduction. In our following result, we drive the best linear unbiased estimator of β (i.e., ˆ β t ) using t observations of rewards and their auxiliary feedback.

Lemma 1. Let t > q + 2 ∈ N and f t be the estimate of function f which uses all information available at the end of round t , i.e., { x s , y s , w s } t s =1 . Then, the best linear unbiased estimator of β ⋆ is

The proof follows after doing some manipulations in Eq. (2) and then using results from linear regression theory. The detailed proof of Lemma 1 and all other missing proofs are given in the supplementary material. After having a new observation of reward and its auxiliary feedback, the best linear unbiased estimator of β ⋆ is re-estimated. If Σ ww or σ yw are known, we can directly use them to estimate β ⋆ by replacing W ⊤ t W t with Σ ww and W ⊤ t Y t with σ yw in Lemma 1. The following result describes the properties of the hybrid reward when β ⋆ is replaced by ˆ β t in Eq. (2).

Theorem 1. Let t > q +2 ∈ N . If ˆ β t as defined in Lemma 1 is used to compute hybrid reward z s,q for any s ≤ t ∈ N , then E [ z s,q ] = f ( x s ) and V ( z s,q ) = ( 1 + q t -q -2 ) (1 -ρ 2 ) σ 2 .

The key takeaways from Theorem 1 are as follows. First, the hybrid reward using ˆ β t is an unbiased estimator of reward function and hence, we can still use it to estimate the reward function f . Second, there is a less reduction in variance (i.e., by a factor ( t -2) / ( t -q -2) of maximum possible variance reduction) when ˆ β t is used for constructing hybrid reward in Eq. (2).

The variance reduction (given in Theorem 1) depends on the auxiliary feedback via two terms: t -2 t -q -2 and ρ 2 (defined in Line 142). Setting q = 1 gives the minimum value for t -2 t -q -2 , but ρ 2 for q = 1 will also be small as it only considers one auxiliary feedback, and hence maximum variance reduction will not be achieved. As we increase the number of auxiliary feedback, ρ 2 increases, leading to more variance reduction. However, the term t -2 t -q -2 increases at the same time, which can negate the variance reduction achieved by smaller ρ 2 . Hence, keeping the number of auxiliary feedback used for hybrid reward small is important. A simple method for selecting a subset of auxiliary feedback (Lavenberg et al., 1982) works as follows: select the auxiliary feedback whose sample correlation coefficient with reward is the largest. Then, select the next auxiliary feedback whose sample partial correlation coefficient with reward was the largest given the first auxiliary feedback selected. Keep repeating the process until there is a variance reduction using additional auxiliary feedback.

3.3 Linear bandits with known auxiliary feedback functions

To highlight the main ideas, we restrict to the linear bandit setting in which the reward and auxiliary feedback functions are linear. In this setting, a learner selects an action x t and observes a reward y t = x ⊤ t θ ⋆ + ε t , where θ ⋆ ∈ R d ( d ≥ 1 ) is unknown and ε t is the zero-mean Gaussian noise with known variance σ 2 . The learner also observes q auxiliary feedback, where i -th auxiliary feedback is denoted by w t,i = x ⊤ t θ ⋆ w,i + ε w t,i . Here, θ ⋆ w,i ∈ R d is known and ε w t,i is the zero-mean Gaussian noise with unknown variance σ 2 w,i . Our goal is to learn a policy that minimizes regret as defined in Eq. (1). We later extend our method for non-linear reward and auxiliary feedback functions in Section 4.3.

Let I be the d × d identity matrix, V t . = ∑ t -1 s =1 x s x ⊤ s , and V t . = V t + λI , where λ > 0 is the regularization parameter that ensures matrix V t is a positive definite matrix. The notation ∥ x ∥ A denotes the weighted l 2 -norm of vector x ∈ R d with respect to a positive definite matrix A ∈ R d × d .

As shown in Theorem 1, the hybrid rewards is an unbiased reward estimator with a smaller variance than observed rewards. Thus, hybrid rewards lead to tighter confidence bounds for parameter θ ⋆ than observed rewards. We propose a simple but effective method to exploit correlated auxiliary feedback, i.e., using hybrid rewards to estimate reward function instead of observed rewards.

Using this method, we adapt the well-known linear bandit algorithm OFUL (Abbasi-Yadkori et al., 2011) to our setting and named this algorithm OFUL-AF. This algorithm works as follows. It takes λ > 0 as input and then initializes V 1 = λI and sets ˆ θ z 1 = 0 R d as initial estimate of parameter θ ⋆ . The superscript ' z ' in ˆ θ z 1 implies that hybrid rewards are used for estimating θ ⋆ . At the beginning of round t , the algorithm selects an action x t that maximizes the upper confidence bound of the action's reward, which is a sum of the estimated reward for the action ( x ⊤ ˆ θ z t ) and a confidence bonus α σ t ∥ x ∥ V -1 t . In the confidence bonus, the first term ( α σ t ) is a slowly increasing function in t whose value is given in Theorem 2, and the second term ( ∥ x ∥ V -1 t ) decreases to zero as t increases.

OFUL-AF Algorithm for Linear Bandits with Auxiliary Feedback

1: Input: λ > 0

2: Initialization: V 1 = λI and ˆ θ z 1 = 0 R d

3: for t = 1 , 2 , . . . do

4: Select action x t = argmax x ∈X ( x ⊤ ˆ θ z t + σα t ∥ x ∥ V -1 t )

6: If t > q +2 , compute upper bound of hybrid reward's sample variance ( ¯ ν z,t ) or else ¯ ν z,t = σ 2

5: Observe reward y t and its auxiliary feedback { w t,i } q i =1

7: If t ≤ q +2 or ¯ ν z,t ≥ σ 2 , set ˆ β t = 0 or else compute ˆ β t using Lemma 1

8: ∀ s ≤ t ∈ N : compute z s,q using ˆ β t in Eq. (2)

10: end for

9: Set V t +1 = V t + x t x ⊤ t , ˆ θ z t +1 = V -1 t +1 ∑ t s =1 x s z s,q

After selecting an action x t , the algorithm observes the reward y t with its associated auxiliary feedback { w t,i } q i =1 . It computes the upper bound on sample variance of hybrid reward (denoted by ¯ ν z,t 1 ) if t > q + 2 or else it is set to σ 2 and then checks two conditions. The first condition (i.e., t ≤ q +2 ) guarantees the sample variance is well-defined. Whereas the second condition (i.e., ¯ ν z,t > σ 2 ) ensures the algorithm at least be as good as OFUL because ¯ ν z,t can be larger than σ 2 due to the overestimation in initial rounds. If both conditions t ≤ q +2 and ¯ ν z,t ≥ σ 2 fail, the value of ˆ β t is re-computed as defined in Lemma 1. The updated ˆ β t is then used to update all hybrid rewards, i.e., z s,q , ∀ s ≤ t ∈ N . Finally, the values of V t +1 and ˆ θ t +1 are updated as V t +1 = V t + x t x ⊤ t and ˆ θ z t +1 = V -1 t +1 ∑ t s =1 x s z s,q , which are then used to select the action in the following round. When ˆ β t = 0 for all hybrid rewards, hybrid rewards are the same as the observed rewards, and hence OFUL-AF is the same as OFUL.

The regret analysis of any bandit algorithm hinges on bounding the instantaneous regret for each action. The following result gives an upper bound on the instantaneous regret of OFUL-AF.

Theorem 2. With a probability of at least 1 -2 δ , the instantaneous regret of OFUL-AF in round t is

The proof follows by bounding the estimation error of the parameter θ ⋆ when the estimation method uses auxiliary feedback. This result shows that auxiliary feedback leads to a better instantaneous regret upper bound and a better regret (as defined in Eq. (1)) than the vanilla OFUL algorithm. Since the improvement in instantaneous regret increase with t , having a single constant to compare with regret of OFUL may lead to weaker regret upper bound than the sum of all instantaneous regret.

4 Estimated auxiliary feedback functions

Auxiliary feedback functions may be unknown in many real-life problems. However, the learner can construct an unbiased estimator for the auxiliary feedback function using historical data or acquiring more samples of auxiliary feedback. But these estimated functions offer a lower variance reduction than known auxiliary functions. To study the effect of using the estimated auxiliary feedback functions on the performance of bandit algorithms, we borrow some techniques from approximate control variate theory (Gorodetsky et al., 2020; Pham and Gorodetsky, 2022) as we discussed next.

4.1 Approximate control variates

Let y be an unbiased estimator of an unknown variable µ and a random variable w with a known estimated mean ( ω e ) be a control variate of y . As long as the known estimated mean is an unbiased estimator of w , one can use it to reduce the variance of y as follows. For any choice of a coefficient β e , define a new random variable as z e . = y -β e ¯ w , where ¯ w = w -ω e . Since ω e is an unbiased estimator of w , it is straightforward to show that z e is also an unbiased estimator of y .

By using properties of variance and covariance, the variance of z e is given by

The variance of z e is minimized by setting β e to β ⋆ e = Cov ( ¯ w, ¯ w ) -1 Cov ( y, ¯ w ) and the minimum value of Var ( z e ) is (1 -ρ 2 e ) Var ( y ) , where ρ e = Cov ( y, ¯ w ) ( Cov ( ¯ w, ¯ w ) -1 / Var ( y ) ) Cov ( y, ¯ w ) .

4.2 Auxiliary feedback with unknown functions as approximate control variates

We now introduce a new definition of hybrid reward that uses estimated auxiliary feedback functions. Let w s,i be the i th auxiliary feedback observed with reward y s and g e,i be the unbiased estimator of function g i . Then, the hybrid reward with q estimated auxiliary feedback functions is defined by

where w s = ( w s, 1 , . . . , w s,q ) , g e,s = ( g e, 1 ( x s ) , . . . , g e,q ( x s )) , and β e = ( β e, 1 , . . . , β e,q ) ⊤ . Let Σ ¯ w ¯ w ∈ R q × q denote the covariance matrix among centered auxiliary feedback (i.e., ¯ w s = w s -g e,s ), and σ y ¯ w ∈ R q × 1 denote the vector of covariance between reward and its centered auxiliary feedback. Then, the variance of z e,s,q is minimized by setting the β e to β ⋆ e = Σ -1 ¯ w ¯ w σ y ¯ w , and the minimum value of Var ( z e,s,q ) is (1 -ρ 2 e ) σ 2 , where ρ 2 e = σ ⊤ y ¯ w Σ -1 ¯ w ¯ w σ y ¯ w /σ 2 .

The definition of hybrid reward given in Eq. (3) is very flexible and allows different estimators to estimate auxiliary feedback functions. The only difference among these estimators is how they partition the available samples of auxiliary feedback to estimate auxiliary function g i . As no optimal partitioning strategy is known, we adopt the Independent Samples (IS) and Multi-Fidelity (MF) sampling strategy for our setting where finite samples of auxiliary feedback are available. Both strategies are proven to be asymptotically optimal (Gorodetsky et al., 2020), implying the variance reduction is asymptotically the same as if auxiliary feedback functions are known.

̸

IS and MF sampling strategy: Let s and s i ⊃ s be the sample sets used for estimating functions f and g i , respectively. Then, for the IS sampling strategy, ( s i \ s ) ∩ ( s j \ s ) = ∅ for i = j , i.e., the extra samples used for estimating the function g i are unique. Whereas, for the MF sampling strategy, s i = s ∪ i j =1 s ′ j and s ′ i ∩ s ′ j = ∅ for i = j , i.e., the estimation of function g i uses the samples that were used for estimating function g i -1 with some additional samples. Refer to Fig. 1 for the visual representation of both sampling strategies. After adopting Theorem 3 and Theorem 4 from

̸

91

92

9q

S

S1

S2

ISSamplingStrategy

91

92

gq

S

S1

S2

MFSamplingStrategy

AFCBandit

Algorithm

Ct

yt

Environment

AFCBandit

Zt,q

Algorithm

Ct

Wt.1

Wt.q

yt

Environment

Gorodetsky et al. (2020) to our setting, we can further simplify Σ ¯ w ¯ w and σ y ¯ w when IS and MF sampling strategies (denoted by e ) are used for estimating auxiliary feedback functions as follows:

where t denotes the number of reward observations with its auxiliary feedback, diag ( A ) represents a vector whose elements are the diagonal of the matrix A , and ◦ denotes an element-wise product. The ij -th element of matrix F e ∈ R q × q is

̸

̸

where r i ∈ R + is the ratio between the total number of samples used for estimating function g i by sampling strategy e and the total number of samples used for estimating f .

Since Σ ¯ w ¯ w and σ y ¯ w may be unknown, they must be estimated to get the best estimate for β ⋆ . Our following result gives the best linear unbiased estimator of β e (i.e., ˆ β e,t ) that uses t observations of rewards and their auxiliary feedback with estimated auxiliary feedback functions.

Lemma 2. Let t > q +2 ∈ N , e is the sampling strategy, and f t be the estimate of function f at the end of round t which uses { x s , y s , w s } t s =1 . Then, the best linear unbiased estimator of β ⋆ e is

where W t is a t × q matrix whose s th row is w s -g e,s and Y t = ( y 1 -f t ( x 1 ) , . . . , y t -f t ( x t )) .

After adopting matrix manipulation tricks from Pham and Gorodetsky (2022) to our setting, the proof follows similar steps as the proof of Lemma 1. We now characterize the properties of the hybrid reward that uses either IS or MF sampling strategy for estimating auxiliary feedback functions.

Theorem 3. Let t > q +2 ∈ N and e is the sampling strategy. If ˆ β e,t as defined in Lemma 2 is used to compute hybrid reward z e,s,q for any s ≤ t ∈ N , then E [ z e,s,q ] = f ( x s ) and V ( z e,s,q ) = ( 1 + a ( e ) q t -q -2 ) (1 -ρ 2 e ) σ 2 , where a ( IS ) = 1 , a ( MF ) = r -1 r if r i = r, ∀ i ∈ { 1 , 2 , . . . , q } when using MF sampling strategy for estimating auxiliary feedback functions.

The key takeaways from Theorem 3 are as follows. First, the hybrid reward with estimated auxiliary feedback is still an unbiased estimator, so one can use it to estimate the reward function f . Second, there is a potential loss in variance reduction as it has an extra multiplicative factor a ( e ) and ρ 2 e ≤ ρ 2 .

Remark 1 . As samples for estimating auxiliary functions increase compared to the reward function, the variance reduction from IS and MF sampling strategy converges to the reduction achieved using known auxiliary functions. As ∀ i : r i →∞ , then F e → 1 q × q . It is now straightforward to see that Σ ¯ w ¯ w will become Σ ww , σ y ¯ w will become σ yw , and hence ρ 2 e = ρ 2 as ∀ i : r i →∞ .

Remark 2 . The IS and MF sampling strategies are shown to be asymptotically optimal (Gorodetsky et al., 2020), i.e., the variance reduction achieved by both strategies is asymptotically the same as if

auxiliary feedback functions are known. However, both sampling strategies are useful in different applications, e.g., the IS sampling strategy suits the problems in which different auxiliary feedback can be independently sampled. In contrast, the MF sampling suits the problems where auxiliary feedback can not be sampled independently.

4.3 Parameterized bandits with estimated auxiliary feedback functions

We now consider the parameterized bandit setting described in Section 2, where the reward and auxiliary feedback function can be non-linear. To exploit the available auxiliary feedback in linear bandits, we propose a method in Section 3.3 that uses hybrid reward in place of rewards to get tight upper confidence bound for the estimator of an unknown reward function and hence smaller regret as compared to the vanilla OFUL due to the smaller variance of the hybrid rewards. We generalize this observation and introduce the notion of Auxiliary Feedback Compatible (AFC) bandit algorithm.

Definition 1 ( AFC Bandit Algorithm ) . Any bandit algorithm A is Auxiliary Feedback Compatible if: (i) A can use correlated reward samples to construct upper confidence bound for reward function and (ii) with probability 1 -δ , its estimated reward function f A t has the following property:

where x ∈ X , σ 2 is the variance of Gaussian noise in observed reward, and O t denotes the observations of actions and their rewards with the parameters of A at the beginning of round t .

As the estimated coefficient vector uses all past samples, the resultant hybrid rewards become correlated due to using this estimated coefficient vector. Bandit algorithms like UCB1 (Auer et al., 2002) and kl-UCB (Cappé et al., 2013) are not AFC as they need independent reward samples to construct upper confidence bounds. In contrast, bandit algorithms like OFUL (Abbasi-Yadkori et al., 2011), Lin-UCB (Chu et al., 2011), UCB-GLM (Li et al., 2017), IGP-UCB, and GP-TS (Chowdhury and Gopalan, 2017) are AFC as they all use techniques proposed in Abbasi-Yadkori et al. (2011) for building the upper confidence bound, which does not need reward samples to be independent.

As AFC bandit algorithms use the noise variance of observed reward for constructing the confidence upper bound, they can also exploit available auxiliary feedback by replacing reward with its respective hybrid reward as shown in Fig. 1 (rightmost figure). We next give an upper bound on the instantaneous regret for any AFC bandit algorithm that uses hybrid rewards instead of observed rewards.

Theorem 4. Let A be an AFC bandit algorithm with | f A t ( x ) -f ( x ) | ≤ σh ( x, O t ) + l ( x, O t ) and ¯ ν e,z,t be the upper bound on sample variance of hybrid reward, whose value is set to σ 2 for t ≤ q +2 . Then, with a probability of at least 1 -2 δ , the instantaneous regret of A after using hybrid rewards (named A -AF ) for reward function estimation in round t is

where e = { IS, MF, KF } , and KF denotes the case where auxiliary functions are known. For t > q +2

After using Theorem 1 and Theorem 3 to replace the variance of hybrid reward, the proof follows similar steps as the proof of Theorem 2. We have given more details about the values of h ( x, O t ) and l ( x, O t ) for different AFC bandit algorithms in Table 1.

5 Experiments

To validate our theoretical results, we empirically demonstrate the performance gain due to auxiliary feedback in different settings of parameterized bandits. We repeat all our experiments 50 times and show the regret as defined in Eq. (1) with a 95% confidence interval (vertical line on each curve shows the confidence interval). Due to space constraints, the details of used problem instances are given in Appendix A.5 of the supplementary material.

Comparing regret with benchmark bandit algorithms: We considered three bandit settings for our experiments: linear bandits, linear contextual bandits, and non-linear contextual bandits (details are given in Appendix A.5). The formal setting of a contextual bandits with auxiliary feedback is given in Appendix A.4. We used the following existing bandit algorithms for these settings: OFUL (Abbasi-Yadkori et al., 2011) for linear bandits, Lin-UCB (Chu et al., 2011) for linear contextual bandits, and Lin-UCB with the polynomial kernel (which we named NLin-UCB) for non-linear contextual bandits. We compare the performance of these benchmark bandit algorithms with four different variants of our algorithms. The first variant assumes the auxiliary feedback functions are known (highlighted by adding '-AF' to the benchmark algorithms). When auxiliary feedback functions are unknown, we use IS and MF sampling strategy while maintaining r = 2 (i.e., getting one extra sample of auxiliary feedback in each round). The IS and MF sampling strategies are the same when only one auxiliary feedback exists. Since we only use one auxiliary feedback in our experiments, we highlight this variant by adding '-IS/MF' to the benchmark algorithms. When IS and MF sampling strategies are used, one needs to update the auxiliary feedback functions in each round to get better estimators. However, it leads to the re-computation of all variables that are needed for updating the hybrid rewards, which is not needed when auxiliary feedback functions are fixed. Therefore, we consider two more computationally efficient variants for the unknown auxiliary functions setting. One variant assumes the knowledge of biased auxiliary feedback, i.e., g i ( x ) + ε g is available instead of g i ( x ) (highlighted by adding '-BE' to the benchmark algorithms). Another variant assumes that some initial samples of auxiliary feedback are available, which are used to get the auxiliary feedback function estimator. We highlight this variant by adding '-EH' to the benchmark algorithms. All variants with given parameters perform better than benchmark bandit algorithms (see Fig. 2a, Fig. 2b, and Fig. 2c). We observe the expected performance among these variants as the variant with a known auxiliary feedback function outperforms all other variants. At the same time, IS/MF sampling strategy-based variant outperforms the other two heuristic variants for the setting of unknown auxiliary feedback function.

Regret vs. different biased estimator: To know the effect of bias in auxiliary feedback (i.e., ε g ) in the mean value of auxiliary feedback, we run an experiment with the same linear contextual bandits experiment setup mentioned above. To see the variation in regret, we set ε g = { 1 , 0 . 2 , 0 . 1 , 0 . 07 , 0 . 05 } . As shown in Fig. 2d, the regret increases with an increase in bias and even starts performing poorly than Lin-UCB. This experiment demonstrates that as long as the bias in auxiliary feedback is within a limit, there will be an advantage to using this computationally efficient variant.

Regret vs. number of historical samples of auxiliary feedback : Increasing the number of historical samples of auxiliary feedback for estimating the auxiliary feedback function reduces the error in its estimation, leading to better performance. To observe this, we use estimators using different numbers of auxiliary feedback samples, i.e., n h = { 5 , 7 , 10 , 15 , 20 } in linear contextual bandits setting. As expected, the regret decreases with an increase in auxiliary feedback samples, but using an estimator with a few samples even performs poorly than Lin-UCB, as shown in Fig. 2e.

Regret vs. correlation coefficient: As theoretical results imply that the regret decreases when the correlation between reward and auxiliary feedback increases. To validate this, we used problem instances with different correlation coefficients in linear contextual bandits setting. As expected, we observe that the regret decreases as the correlation coefficient increases, as shown in Fig. 2f.

6 Conclusion

This paper studies a novel parameterized bandit problem in which a learner observes auxiliary feedback correlated with the observed reward. We first introduce the notion of 'hybrid reward,' which combines the reward and its auxiliary feedback. To get the maximum benefit from hybrid reward, we treat auxiliary feedback as a control variate and then extend control variate theory to a setting

(a) Linear Bandit

(b) Linear Contextual Bandits

(c) Non-linear Contextual Bandits

(d) Biased Estimator

(e) Estimator from historical data

(f) Influence of correlation

175

40

250

150

200

gret

125

egret

30

gret

6

100

20

6

150

R

75

OFUL

Lin-UCB

100

NLin-UCB

50

OFUL-EH (n = 10)

10

NLin-UCB-EH (n, = 10)

25

OFUL-BE (εg = 0.1)

OFUL-IS/MF (r= 2)

50

NLin-UCB-BE (ε。 = 0.1)

NLin-UCB-IS/MF (r = 2)

0

OFUL-AF

0

0

NLin-UCB-AF

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

Rounds

Rounds

Rounds

Lin-UCB

Lin-UCB

40

40

80

Regret

30

egret

30

Regret

60

20

20

40

10

10

20

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

Rounds

Rounds

Rounds

where a function can parameterize control variates. Equipped with these results, we show that the variance of hybrid rewards is smaller than observed rewards. We then use these hybrid rewards to estimate the reward function, leading to tight confidence bounds and hence smaller regret. We have proved that the expected instantaneous regret of any AFC bandit algorithm after using hybrid rewards is improved by a factor of O ((1 -ρ 2 ) 1 2 ) , where ρ is the correlation coefficient of the reward and its auxiliary feedback. Our experiments also validate our theoretical results. An interesting future direction is to extend these results to bandit settings with heteroscedastic and non-Gaussian noise.

Acknowledgments and Disclosure of Funding

DesCartes: this research is supported by the National Research Foundation, Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.

References

Aleksandrs Slivkins. Introduction to Multi-Armed Bandits. Foundations and Trends® in Machine Learning , 2019.

Tor Lattimore and Csaba Szepesvári. Bandit Algorithms . Cambridge University Press, 2020.

Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A Contextual-Bandit Approach to Personalized News Article Recommendation. In Proc. WWW , pages 661-670, 2010.

Wei Chu, Lihong Li, Lev Reyzin, and Robert E Schapire. Contextual Bandits with Linear Payoff Functions. In Proc. AISTATS , pages 208-214, 2011.

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved Algorithms for Linear Stochastic Bandits. In Proc. NeurIPS , pages 2312-2320, 2011.

Shipra Agrawal and Navin Goyal. Thompson Sampling for Contextual Bandits with Linear Payoffs. In Proc. ICML , pages 127-135, 2013.

Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric Bandits: The Generalized Linear Case. In Proc. NeurIPS , pages 586-594, 2010.

Lihong Li, Yu Lu, and Dengyong Zhou. Provably Optimal Algorithms for Generalized Linear Contextual Bandits. In Proc. ICML , pages 2071-2080, 2017.

Kwang-Sung Jun, Aniruddha Bhargava, Robert Nowak, and Rebecca Willett. Scalable Generalized Linear Bandits: Online Computation and Hashing. In Proc. NeurIPS , pages 99-109, 2017.

Michal Valko, Nathan Korda, Rémi Munos, Ilias Flaounas, and Nello Cristianini. Finite-time Analysis of Kernelised Contextual Bandits. In Proc. UAI , pages 654-663, 2013.

Sayak Ray Chowdhury and Aditya Gopalan. On Kernelized Multi-armed Bandits. In Proc. ICML , pages 844-853, 2017.

Noga Alon, Nicolo Cesa-Bianchi, Ofer Dekel, and Tomer Koren. Online Learning with Feedback Graphs: Beyond Bandits. In Proc. COLT , pages 23-35, 2015.

Yifan Wu, András György, and Csaba Szepesvári. Online Learning with Gaussian Payoffs and Side Observations. In Proc. NeurIPS , pages 1360-1368, 2015.

Stephen S Lavenberg and Peter D Welch. A Perspective on the Use of Control Variables to Increase the Efficiency of Monte Carlo Simulations. Management Science , pages 322-335, 1981.

Stephen S Lavenberg, Thomas L Moeller, and Peter D Welch. Statistical Results on Control Variables with Application to Queueing Network Simulation. Operations Research , pages 182-202, 1982.

Arun Verma and Manjesh K Hanawal. Stochastic Multi-Armed Bandits with Control Variates. In Proc. NeurIPS , pages 27592-27603, 2021.

Barry L Nelson. Control Variate Remedies. Operations Research , pages 974-992, 1990.

Julia Kreutzer, Artem Sokolov, and Stefan Riezler. Bandit Structured Prediction for Neural Sequence-to-Sequence Learning. In Proc. ACL , pages 1503-1513, 2017.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Nikos Vlassis, Ashok Chandrashekar, Fernando Amat Gil, and Nathan Kallus. Control Variates for Slate Off-Policy Evaluation. In Proc. NeurIPS , pages 3667-3679, 2021.

Stephane Caron, Branislav Kveton, Marc Lelarge, and S Bhagat. Leveraging Side Observations in Stochastic Bandits. In Proc. UAI , pages 142-151, 2012.

Shie Mannor and Ohad Shamir. From Bandits to Experts: On the Value of Side-Observations. In Proc. NeurIPS , pages 684-692, 2011.

Tomáš Kocák, Gergely Neu, Michal Valko, and Rémi Munos. Efficient learning by implicit exploration in bandit problems with side observations. In Proc. NeurIPS , pages 613-621, 2014.

Noga Alon, Nicolo Cesa-Bianchi, Claudio Gentile, Shie Mannor, Yishay Mansour, and Ohad Shamir. Nonstochastic Multi-Armed Bandits with Graph-Structured Feedback. SIAM Journal on Computing , pages 1785-1826, 2017.

Arun Verma, Manjesh K Hanawal, Csaba Szepesvári, and Venkatesh Saligrama. Online Algorithm for Unsupervised Sensor Selection. In Proc. AISTATS , pages 3168-3176, 2019.

Arun Verma, Manjesh K Hanawal, and Nandyala Hemachandra. Thompson Sampling for Unsupervised Sequential Selection. In Proc. ACML , pages 545-560, 2020a.

Arun Verma, Manjesh K Hanawal, Csaba Szepesvári, and Venkatesh Saligrama. Online Algorithm for Unsupervised Sequential Selection with Contextual Information. In Proc. NeurIPS , pages 778-788, 2020b.

A. P. James. Variance Reduction Techniques. Journal of the Operational Research Society , pages 525-530, 1985.

Barry L Nelson. Batch size effects on the efficiency of control variates in simulation. European Journal of Operational Research , pages 184-196, 1989.

Zdravko Botev and Ad Ridder. Variance Reduction. Wiley statsRef: Statistics reference online , pages 1-6, 2017.

Yutian Chen and Zoubin Ghahramani. Scalable Discrete Sampling as a Multi-Armed Bandit Problem. In Proc. ICML , pages 2492-2501, 2016.

Alex A Gorodetsky, Gianluca Geraci, Michael S Eldred, and John D Jakeman. A Generalized Approximate Control Variate Framework for Multifidelity Uncertainty Quantification. Journal of Computational Physics , page 109257, 2020.

Trung Pham and Alex A Gorodetsky. Ensemble Approximate Control Variate Estimators: Applications to MultiFidelity Importance Sampling. Journal on Uncertainty Quantification , pages 1250-1292, 2022.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time Analysis of the Multiarmed Bandit Problem. Machine Learning , pages 235-256, 2002.

Olivier Cappé, Aurélien Garivier, Odalric-Ambrym Maillard, Rémi Munos, and Gilles Stoltz. Kullback-Leibler Upper Confidence Bounds for Optimal Sequential Allocation. The Annals of Statistics , pages 1516-1541, 2013.

F Hayashi. Econometrics . Princeton University Press, 2000.

Sara A Van De Geer. Estimation. Encyclopedia of Statistics in Behavioral Science , pages 549-553, 2005.

Bruce Schmeiser. Batch Size Effects in the Analysis of Simulation Output. Operations Research , pages 556-568, 1982.

A Supplementary material

A.1 Missing proofs related to auxiliary feedback

Results from linear regression

We first state results for linear regression that we will use in the subsequent proofs. Consider the following regression problem with t samples and q features:

where z s ∈ R is the s th target variable, x s = ( x s 1 , . . . , x sq ) ∈ R q is the s th feature vector, θ ∈ R q is the unknown regression parameters, and ε s is a normally distributed noise with mean 0 and constant variance σ 2 . The values of noise ε s form an IID sequence and are independent of x s . Let

Then, the best linear unbiased estimator of θ is ˆ θ t = ( X ⊤ t X t ) -1 X ⊤ t Z t , which has the following finite sample properties.

Fact 1 . The following are the finite sample properties of the least square estimator ˆ θ t :

where ( X ⊤ t X ) -1 ii is the ii -element of the matrix ( X ⊤ t X ) -1 .

In the above result, the first two properties are from Proposition 1.1 of Hayashi (2000), whereas the third property is from Van De Geer (2005). The following result gives the finite sample properties of the estimator of noise variance σ 2 .

Fact 2 . (Hayashi, 2000, Proposition 1.2) Let ˆ σ 2 t = 1 t -q ∑ t s =1 ( z s -x ⊤ s ˆ θ t ) 2 be estimator of σ 2 and t > q (so that ˆ σ 2 t is well defined). Then, ˆ σ 2 t is an unbiased estimator of σ 2 , i.e., E [ ˆ σ 2 t | X t ] = σ 2 .

Using the Schur complement, we have the following results about the inverse of the block matrix.

Fact 3 . Let G = ( t B C D ) be a block matrix, where t ∈ R \ { 0 } , B , C , D are respectively 1 × q , q × 1 , and q × q matrices of real numbers. Then, G -1 11 = t -1 + t -1 B ( tD -CB ) -1 C .

Control variates theory

Let y be the random variable of interest with unknown mean µ . There are q control variates correlated with y , where i th control variate has mean ω i and its s th observation is denoted by w s,i . For any s ∈ { 1 , . . . , t } , we define a variable z s using s th observation of y s and its control variates as follows:

where w s = ( w s, 1 , . . . , w s,q ) and ω = ( ω 1 , . . . , ω q ) . The above equation can be re-written as:

Under the assumption of z s being a unbiased estimator of µ , we can write y s as follows:

where ε z, 1 , . . . , ε z,t are IID and have zero mean Gaussian noise with variance (1 -ρ 2 ) σ 2 . Let

The best linear unbiased estimator of γ is ˆ γ = ( W ⊤ t W t ) -1 W ⊤ t Y t . To get ˆ µ z,t and ˆ β ⋆ , we expand ˆ γ as follows:

After taking first matrix from RHS to LHS and using ˆ γ = ( ˆ µ z,t ˆ β t ) , we have

From above, we get the following equation:

Similarly, we have another equation as follows:

In the following, we first state the fundamental results from the control variates theory. Fact 4 . (Nelson, 1990, Theorem 1) Let O s = ( Y s , W s, 1 , . . . , W s,q ) ⊤ follow a ( q +1) -variate normal distribution with mean vector ( µ, ω ) and { O 1 , . . . , O t } be a IID sequence. Assume ˆ µ z,t = ∑ t s =1 z s , where z s = y s -( w s -ω ) ˆ β t and ˆ β t used here is given by Eq. (6), then

where σ Y W Σ -1 WW σ ⊤ Y W /σ 2 is the square of the multiple correlation coefficient, σ 2 = V ( Y ) , and σ Y W = ( Cov ( Y, W 1 ) , . . . , Cov ( Y, W q )) (we have dropped the subscript s as observations are IID).

Auxiliary feedback as control variates

Lemma 1. Let t > q + 2 ∈ N and f t be the estimate of function f which uses all information available at the end of round t , i.e., { x s , y s , w s } t s =1 . Then, the best linear unbiased estimator of β ⋆ is

where W t is a t × q matrix whose s th row is ( w s -g s ) and Y t = ( y 1 -f t ( x 1 ) , . . . , y t -f t ( x t )) .

Proof. Recall Eq. (2) for s th hybrid reward with known auxiliary functions, i.e., z s,q = y s -( w s -g s ) β , which can be re-written as y s = z s,q + ( w s -g s ) β . By definition, z s,q = f ( x s ) + ε z s for optimal β , where ε z,s is zero-mean Gaussian noise with variance (1 -ρ 2 ) σ 2 . Then, y s = f ( x s ) + ( w s -g s ) β + ε z,s . Let φ be an unknown function that maps every x to a space where f ( x ) = φ ( x ) ⊤ f holds. Then we can re-write the above equation as follows:

Adapting Eq. (4) to our setting, we have

Let f t is the estimated f using available information, i.e., { x s , y s , w s } t s =1 . To get best linear unbiased estimator for β ⋆ , we use the following equation from above matrix,

Since the reward and its auxiliary feedback observations are functions of the selected action, we can not directly use the control variate theory due to parameterized mean values of the reward and its auxiliary feedback. To overcome this challenge, we centered the observations by its function value and defined new centered variables as follows:

In our setting, these centered variables ( y c s , w c s , and z c s , ) follow zero mean Gaussian distributions with variance σ 2 , σ 2 w = ( σ 2 w, 1 , . . . , σ 2 w,q ) , and (1 -ρ 2 ) σ 2 , respectively.

Theorem 1. Let t > q +2 ∈ N . If ˆ β t as defined in Lemma 1 is used to compute hybrid reward z s,q for any s ≤ t ∈ N , then E [ z s,q ] = f ( x s ) and V ( z s,q ) = ( 1 + q t -q -2 ) (1 -ρ 2 ) σ 2 .

Proof. The sequence ( y c s , w c s ) t s =1 is an IID sequence and follows a Gaussian distribution with mean 0 . We now define z c s,q = y c s -w c s β = y c s -( w s -g s ) β , which can be re-written as y c s,q = z c s +( w s -g s ) β . Let f t be the estimated f using available information, i.e., { x s , y s , w s } t s =1

and hence we can write estimated z s,q as ˆ z s,q = f t ( x s ) and hence ˆ z c s,q = f t ( x s ) -f ( x s ) . Now, adapting Eq. (6) to our setting and replacing estimated mean in Y t by ˆ z c s,q , s th value of Y t is y c s -ˆ z c s,q = y s -f ( x s ) -( f t ( x s ) -f ( x s )) = y s -f t ( x s ) . With these manipulations, we get the following best linear unbiased estimator for β ⋆ :

which is the same as defined in Lemma 1.

By adapting Fact 4 for a single sample (i.e., z s,q ) while using ˆ β t to define hybrid reward, we have

By extending the definition of E [ z c s,q ] we have,

This proofs the hybrid reward is an unbiased estimator of reward.

Since variance is invariant to constant change, we have

Lemma 2. Let t > q +2 ∈ N , e is the sampling strategy, and f t be the estimate of function f at the end of round t which uses { x s , y s , w s } t s =1 . Then, the best linear unbiased estimator of β ⋆ e is

where W t is a t × q matrix whose s th row is w s -g e,s and Y t = ( y 1 -f t ( x 1 ) , . . . , y t -f t ( x t )) .

Proof. Recall the s th hybrid reward defined in Eq. (3) using sampling strategy e as z e e,s,q = y s -( w s -g e,s ) β e , which can be re-written as y s = z e e,s,q +( w s -g e,s ) β e . Following similar steps as of Lemma 1, we can re-write the above equation as y s = f ⊤ φ ( x s ) + ( w s -g e,s ) β e + ε z,s .

From Appendix D and E of Gorodetsky et al. (2020), we have W ⊤ e,t W e,t = W ⊤ t W t ◦ F e and W ⊤ e,t Y t = diag ( F e ) ◦ W ⊤ t Y t . Using these two equality, we have

Theorem 3. Let t > q +2 ∈ N and e is the sampling strategy. If ˆ β e,t as defined in Lemma 2 is used to compute hybrid reward z e,s,q for any s ≤ t ∈ N , then E [ z e,s,q ] = f ( x s ) and V ( z e,s,q ) = ( 1 + a ( e ) q t -q -2 ) (1 -ρ 2 e ) σ 2 , where a ( IS ) = 1 , a ( MF ) = r -1 r if r i = r, ∀ i ∈ { 1 , 2 , . . . , q } when using MF sampling strategy for estimating auxiliary feedback functions.

Proof. The proof follows the similar steps as Theorem 1 except we adapt the part (b.) of Theorem 4 from Pham and Gorodetsky (2022) instead of using Fact 4 to show the variance reduction when sampling strategy (IS or MF) is used for estimating auxiliary feedback functions.

A.2 Unbiased estimate of variance

Consider the following regression problem with target variable y s , which is defined as follows:

where ε z, 1 , . . . , ε z,t are IID and have zero mean Gaussian noise with variance (1 -ρ 2 ) σ 2 . Let

Now, using Fact 1, we have V (ˆ µ z,t ) = σ 2 ( W ⊤ t W t ) -1 11 , where ( Y ⊤ Y ) -1 11 is the upper left most element of matrix ( Y ⊤ Y ) -1 (Schmeiser, 1982). Then after t observations, the unbiased variance estimator of V (ˆ µ z,t ) is given by

where ˆ σ 2 z,t = 1 t -q -1 ∑ t s =1 ( y s -ˆ µ z,t ) 2 (Nelson, 1990), which is also an unbiased variance estimator of σ 2 (from Fact 2). Further, ˆ ν z,t is also an unbiased estimator of V (ˆ µ z,t ) , i.e., E [ˆ ν z,t ] = V (ˆ µ z,t ) (Nelson, 1990, Theorem 1). We can adapt this approach to our setting. However when noise variance ( σ ) is unknown, computing ( W ⊤ t W t ) -1 11 may not be possible to general function f as φ function may not be known. Though the setting in which ( W ⊤ t W t ) -1 11 can be computed, we have to use the upper bound of variance to construct the confidence bound for reward function f as random sample variance estimate can be small and leads to invalid confidence bounds. Given t observations, the upper bound of the sample variance is given by ¯ ν z,t = ( t -2)ˆ ν z,t χ 2 1 -δ,t , where χ 2 1 -δ,t denotes 100(1 -δ ) th percentile value of the chi-squared distribution with t -2 degrees of freedom.

A.3 Missing proofs related to regret analysis

Theorem 2. With a probability of at least 1 -2 δ , the instantaneous regret of OFUL-AF in round t is

where α σ t = √ min( σ 2 , ¯ ν z,t -1 ) α t , ∥ θ ⋆ ∥ 2 ≤ S , and α t = √ d log ( 1+ tL 2 /λ δ ) . For t > q +2 and ¯ ν z,t < σ 2 , E [ r t ( OFUL-AF )] ≤ ˜ O ( ( ( t -3)(1 -ρ 2 ) t -q -3 ) 1 2 r t ( OFUL ) ) . Here, ˜ O hides constant terms.

Proof. When only observed rewards are used for estimating underlying unknown parameters in the linear bandit setting, i.e., ˆ θ t = V -1 t ∑ t s =1 x s y s , then with probability 1 -δ , the confidence bound (Abbasi-Yadkori et al., 2011, Theorem 1) is

where σ 2 is the variance of observed rewards given action (since the noise variance is σ 2 ). To ensure the performance of OFUL-AF is as good as OFUL, we only used hybrid reward samples for estimation when the upper bound on the variance of hybrid rewards is smaller than the variance of rewards, i.e., ¯ ν z,t -1 < σ 2 . At the beginning of round t , the variance upper bound of hybrid rewards is computed using t -1 observations and given by ¯ ν z,t -1 = ( t -2)ˆ ν z,t -1 χ 2 1 -δ,t , where ˆ ν z,t -1 is an unbiased sample variance estimate of hybrid rewards using t -1 observations and χ 2 1 -δ,t (implying the variance upper bound holds with at least probability of 1 -δ ) denotes 100(1 -δ ) th percentile value of the chi-squared distribution with t -2 degrees of freedom. When ¯ ν z,t < σ 2 , we replace rewards { y s } t s =1 with its respective hybrid rewards, i.e., { z s,q } t s =1 to estimate underlying parameter and use it for next round. After using hybrid rewards to estimate the unknown parameter, we replace

σ 2 in Eq. (7) with the variance upper bound of hybrid rewards. Then we get the following upper bound which holds with a probability of 1 -2 δ .

where α σ t = √ min( σ 2 , ¯ ν z,t -1 ) α t and α t = √ d log ( 1+ tL 2 /λ δ ) .

Let action x t be selected in the round t . Then, the instantaneous regret is given as follows:

A sub-optimal action is only selected when its upper confidence bound is larger than the optimal action. Then, if ∥ ∥ ∥ ˆ θ t -θ ⋆ ∥ ∥ ∥ V t = α σ t + λ 1 / 2 S , then we have

Let X t = { x s } t s =1 . For t > q +2 and ¯ ν z,t < σ 2 , the expected instantaneous regret of OFUL-AF is

Since ˆ ν z,t -1 is an unbiased estimator of the sample variance of hybrid rewards, E [ˆ ν z,t -1 ] = V ( z s,q ) for s ∈ { 1 , . . . , t } . Using Theorem 1, we have V ( z s,q ) = ( 1 + q t -q -3 ) (1 -ρ 2 ) σ 2 =

( ( t -3)(1 -ρ 2 ) t -q -3 ) σ 2 as t th observation is not available at the beginning of the round t . With increasing t , C t = √ ( t -2) χ 2 1 -δ,t tends to 1. With all these observations, we have

Let r t ( OFUL ) be the upper bound on instantaneous regret for OFUL algorithm, i.e., r t ( OFUL ) = 2 ( σα t + λ 1 / 2 S ) ∥ x t ∥ V -1 t . Then, we have

Theorem 4. Let A be an AFC bandit algorithm with | f A t ( x ) -f ( x ) | ≤ σh ( x, O t ) + l ( x, O t ) and ¯ ν e,z,t be the upper bound on sample variance of hybrid reward, whose value is set to σ 2 for t ≤ q +2 . Then, with a probability of at least 1 -2 δ , the instantaneous regret of A after using hybrid rewards (named A -AF ) for reward function estimation in round t is

where e = { IS, MF, KF } , and KF denotes the case where auxiliary functions are known. For t > q +2 and ¯ ν e,z,t < σ 2 , E [ r t ( A -AF )] ≤ ˜ O ( (( t -(1 -a ( e )) q -3 t -q -3 ) (1 -ρ 2 e ) ) 1 2 r t ( A ) ) , where a ( KF ) = 1 .

Proof. Let A be an AFC bandit algorithm with | f A t ( x ) -f ( x ) | ≤ σh ( x, O t ) + l ( x, O t ) and ¯ ν e,z,t be the upper bound on sample variance of hybrid reward. After A uses hybrid rewards for estimating function f , then, with probability at least 1 -2 δ ,

The proof follows similar steps as the first part of the proof of Theorem 2. The only key difference is the upper bound of variance of hybrid rewards, which depends on the underlying sampling strategy based on whether auxiliary functions are known or unknown. The upper bound on sample variance is given by ¯ ν e,z,t = ( t -2)ˆ ν e,z,t -1 χ 2 1 -δ,t , where ˆ ν e,z,t -1 is an unbiased sample variance estimate of hybrid rewards using t -1 observations with sampling strategy e and χ 2 1 -δ,t (implying the variance upper bound holds with at least probability of 1 -δ ) denotes 100(1 -δ ) th percentile value of the chi-squared distribution with t -2 degrees of freedom.

Let action x t be selected in the round t . Then, the instantaneous regret is given as follows:

in which the first and last inequalities have used the upper bound given in Eq. (8), and the second inequality follows because actions are selected using the upper confidence bounds. The remaining proof will follow the similar steps as the second part of Theorem 2 except using Theorem 3 instead of Theorem 1 for quantifying the variance reduction due to hybrid rewards when IS or MF sampling strategy is used for estimating auxiliary feedback function.

A.4 Auxiliary feedback in contextual bandits

Many real-life applications have some additional information readily available for the learner before selecting an action, e.g., users' profile information is known to the online platform before making any recommendations. Such information is treated as contextual information in bandit literature, and the bandit problem having contextual information is refereed as contextual bandits (Li et al., 2010). Since the value of the reward function also depends on the context, the learner's goal is to use contextual information to select a better action.

We extend our results for the contextual bandits problem. In this setting, we assume that a learner has been given an action set denoted by A . In round t , the environment generates a vector ( x t,a , y t,a , { w t,a,i } q i =1 ) for each action a ∈ A . Here, x t,a is the context-action d -dimensional feature vector of observed context in round t and action a , y t,a is the stochastic reward received for context-action pair x t,a , and w t,a,i is the i th auxiliary feedback associated with the reward y t,a . We assume that the reward is a function of the context-action pair x t,a , which is given as y t,a = f ( x t,a ) + ε t , where f : R d → R is an unknown function and ε t is a zero-mean Gaussian noise with variance σ 2 . The auxiliary feedback is also assumed to be a function of the context-action pair x t,a , given as W t,a,i = g i ( x t,a ) + ε w t,i , where g i : R d → R and ε w t,i is a zero-mean Gaussian noise with variance σ 2 w . The correlation coefficient between reward and associated auxiliary feedback is denoted by ρ .

We denote the optimal action for a context observed in the round t as a ⋆ t = argmax a ∈A f ( x t,a ) . The interaction between a learner and its environment is given as follows. At the beginning of round t , the environment generates a context, and then the learner selects an action a t from action set A for that context using past information of context-actions feature vector, observed rewards and its associated auxiliary feedback until round t -1 . After selecting action a t , the learner receives a reward ( y t,a t ) with its associated auxiliary feedback and incurs a penalty (or instantaneous regret) r t , where r t = f ( x t,a ⋆ t ) -f ( x t,a t ) . We aim to learn a sequential policy that selects actions to minimize the total penalty and evaluate the performance of such policy through regret , which is the sum of the penalty incurred by the learner. Formally, for T contexts, the regret of a policy π that selects action a t for a context observed in round t is given by

A policy π is a good policy when it has sub-linear regret. This implies that the policy will eventually learn to recommend the best action for every context. Similar to the parameterized bandit problem case, we can use the existing contextual bandit algorithms, which are AFC bandit algorithms. Depending on the problem, an appropriate AFC contextual bandit algorithm is selected that uses hybrid rewards to estimate reward function. The smaller variance of hybrid rewards leads to tighter upper confidence bound of the unknown reward function and hence smaller regret.

A.5 More details about experiments

To demonstrate the performance gain from using auxiliary feedback, we have considered three different bandit settings: linear bandits, linear contextual bandits, and non-linear contextual bandits. The details of the problem instance used in our experiments are as follows.

(a) Biased Estimator

(b) Estimator from historical data

(c) Influence of correlation

(d) Biased Estimator

(e) Estimator from historical data

(f) Influence of correlation

175

150

OFUL

OFUL-BE (Eg = 1)

OFUL-BE (Eg =0.2)

OFUL-BE (Eg = 0.1)

150

175

OFUL-EH（nh=7)

OFUL-EH(nn=10)

OFUL

OFUL-EH (n = 5)

700

600

OFUL-AF (p² = 0.1)

OFUL-AF (o² = 0.2)

OFUL-AF (o² = 0.5)

OFUL-AF (p²= 0.3)

Regret

125

OFUL-BE（E=0.07)

OFUL-BE(E=0.05)

egret

125

OFUL-EH (n = 20)

OFUL-EH (n = 15)

Regret

500

OFUL-AF (p² = 0.7)

100

OFUL-AF

100

OFUL-AF

400

75

75

300

50

50

200

25

25

100

0

0

0

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

Rounds

Rounds

Rounds

NLin-UCB

NLin-UCB

500

NLin-UCB-AF (p² = 0.1)

250

NLin-UCB-BE (Eg = 1)

250

NLin-UCB-EH (n = 5)

NLin-UCB-AF (p² = 0.2)

NLin-UCB-BE (Eg = 0.2)

NLin-UCB-EH (nn = 7)

400

NLin-UCB-AF (p² = 0.3)

egret

200

150

+

+一

NLin-UCB-BE (Eg = 0.07)

NLin-UCB-BE (Eg = 0.05)

NLin-UCB-BE (Eg = 0.1)

NLin-UCB-AF

egret

200

150 -

-

NLin-UCB-EH (nn = 10)

NLin-UCB-EH (n = 15)

NLin-UCB-AF

NLin-UCB-EH (n= 20)

egret

北

300

NLin-UCB-AF (p² = 0.5)

NLin-UCB-AF (o² = 0.7)

R

100

R

100

R

200

50

50

100

D

0

0

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

Rounds

Rounds

Rounds

Linear bandits: We use a 5 -dimensional space in which each sample is represented by x = ( x 1 , . . . , x 5 ) , where the value of x j is restricted in ( -3 , 3) . We randomly select a 5 -dimensional vector θ ⋆ with a unit norm whose each value is restricted in (0 , 1) . In all linear bandits experiments, we use λ = 0 . 01 , L = 2 . 236 , S = 1 , and δ = 0 . 05 . In round t , the reward for selected action x t is

where v t = x ⊤ t θ ⋆ v + ε v t and w t = x ⊤ t θ ⋆ w + ε w t . Weset θ ⋆ v = (0 , θ ⋆ 2 , 0 , θ ⋆ 4 , 0) and θ ⋆ w = ( θ ⋆ 1 , 0 , θ ⋆ 3 , 0 , θ ⋆ 5 ) . As we treat w t as auxiliary feedback, θ ⋆ w may be assumed to be known in some experiments. The random noise ε v t is zero-mean Gaussian noise with variance σ 2 v . Whereas ε w t is also zero-mean Gaussian noise, but the variance is σ 2 w . We assumed that σ 2 = σ 2 v + σ 2 w is known, but not the σ 2 v and σ 2 w . The default value of σ 2 v = 0 . 01 and σ 2 w = 0 . 01 . It can be easily shown that the correlation coefficient of y t and w t is ρ = √ σ 2 w / ( σ 2 v + σ 2 w ) . We run each experiment for 5000 rounds.

Linear contextual bandits: We first generate a 2 -dimensional synthetic dataset with 5000 data samples. Each sample is represented by x = ( x 1 , x 2 ) , where the value of x j is drawn uniformly at random from ( -1 , 1) . Our action set A has four actions: { ( x 1 , x 2 ) , ( x 1 , -x 2 ) , ( -x 1 , x 2 ) , ( -x 1 , -x 2 ) } . We uniformly generate a θ ⋆ such that its norm is 1 . In all experiments, the data samples are treated as contexts, and we use λ = 0 . 01 , L = 1 . 41 , S = 1 , and δ = 0 . 05 . The observed reward for a context-action feature vector has two components. We treated one of the components as auxiliary feedback. In round t , the reward context-action feature vector x t,a is given as follows:

where v t,a t = x ⊤ t,a θ ⋆ v + ε v t and w t,a t = x ⊤ t,a θ ⋆ w + ε w t . Weset θ ⋆ v = (0 , θ ⋆ 2 , 0 , θ ⋆ 4 ) and θ ⋆ w = ( θ ⋆ 1 , 0 , θ ⋆ 3 , 0) . As we treat w t,a t as auxiliary feedback, θ ⋆ w is known for some experiments. The random noise ε v t is zero-mean Gaussian noise with variance σ 2 v . Whereas ε w t is also zero-mean Gaussian noise, but the variance is σ 2 w . We assumed that σ 2 = σ 2 v + σ 2 w is known, but not the σ 2 v and σ 2 w . The default value of σ 2 v = 0 . 01 and σ 2 w = 0 . 01 . It can be easily shown that the correlation coefficient of y t,a and w t,a is ρ = √ σ 2 w / ( σ 2 v + σ 2 w ) .

(a) Linear Bandit

(b) Linear Contextual Bandits

(c) Non-linear Contextual Bandits

OFUL

Lin-UCB

NLin-UCB

175

OFUL-EH (n = 5)

40

250

NLin-UCB-EH (nn = 5)

OFUL-EH(n=7)

NLin-UCB-EH (n = 7)

150

OFUL-EH(n=10)

NLin-UCB-EH (n = 10)

egret

125

OFUL-EH (n= 20)

OFUL-EH (n = 15)

Regret

30

egret

200

NLin-UCB-EH (n = 20)

NLin-UCB-EH (n = 15)

100

OFUL-AF

150

NLin-UCB-AF

Reg

20

Reg

75

100

50

10

25

50

0

0

0

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

0

1000

2000

3000

4000

5000

Rounds

Rounds

Rounds

Non-linear contextual bandits: This problem instance is adapted from the linear contextual bandits problem instance. We first generate a 2 -dimensional synthetic dataset with 5000 data samples. Each sample is represented by x = ( x 1 , x 2 ) , where the value of x j is drawn uniformly at random from ( -1 , 1) . We then use a polynomial kernel with degree 2 to have a non-linear transformation of samples. We removed (i.e., bias) the first (i.e., 1 ) and last value ( i.e., x 2 2 ) from the transformed samples, which reduced the dimensional of each transformed sample to 4 and represented as ( x 1 , x 2 , x 2 1 , x 1 x 2 ) , which is used as context. For this setting, the action set A has six actions: { ( x 1 , x 2 , -x 2 1 , -x 1 x 2 ) , ( x 1 , -x 2 , x 2 1 , -x 1 x 2 ) , ( -x 1 , x 2 , x 2 1 , -x 1 x 2 ) , ( x 1 , -x 2 , -x 2 1 , x 1 x 2 ) , ( -x 1 , x 2 , -x 2 1 , x 1 x 2 ) , ( -x 1 , -x 2 , x 2 1 , x 1 x 2 ) , } . We uniformly generate a θ ⋆ such that its norm is 1 . In all experiments, we use λ = 0 . 01 , L = 2 , S = 1 , and δ = 0 . 05 . The observed reward for a context-action feature vector has two components. We treated one of the components as auxiliary feedback. In round t , the reward context-action feature vector x t,a is given as follows:

where v t,a t = x ⊤ t,a θ ⋆ v + ε v t and w t,a t = x ⊤ t,a θ ⋆ w + ε w t . We set θ ⋆ v = (0 , θ ⋆ 2 , 0 , θ ⋆ 4 , 0 , θ ⋆ 6 , 0 , θ ⋆ 8 ) and θ ⋆ w = ( θ ⋆ 1 , 0 , θ ⋆ 3 , 0 , θ ⋆ 5 , 0 , θ ⋆ 7 , 0) . As we treat w t,a t as auxiliary feedback, θ ⋆ w is known for some experiments. The random noise ε v t is zero-mean Gaussian noise with variance σ 2 v . Whereas ε w t is also zero-mean Gaussian noise, but the variance is σ 2 w . We assumed that σ 2 = σ 2 v + σ 2 w is known, but not the σ 2 v and σ 2 w . The default value of σ 2 v = 0 . 01 and σ 2 w = 0 . 01 . It can be easily shown that the correlation coefficient of y t,a and w t,a is ρ = √ σ 2 w / ( σ 2 v + σ 2 w ) .

Regret with varying correlation coefficient: As the correlation coefficient of reward and auxiliary feedback is ρ = √ σ 2 w / ( σ 2 v + σ 2 w ) , we varied σ v over the values { 0 . 3 , 0 . 2 , 0 . 1528 , 0 . 1 , 0 . 0655 } to obtain problem instances with different correlation coefficient for all problem instances.

Variance estimation: Since the value of σ 2 is know in all our experiments, we directly estimate the correlation coefficient ( ρ ) as ˆ ρ = Cov ( y, w ) / ( √ V ( w ) σ ) . Then, use it to set ¯ ν e,z,t = (1 -ˆ ρ 2 ) σ 2 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors studied a variant of the parameterized bandits problem where the learner has access to auxiliary feedback that is correlated with the observed reward. The authors proposed a method that leverages the auxiliary feedback to construct a reward estimator with more accurate confidence bounds, resulting in better regret bounds. The paper provides a characterization of the regret reduction in terms of the correlation coefficient between the reward and auxiliary feedback. Finally, they demonstrated the effectiveness of their method via numerical experiments.\n\nSTRENGTHS:\n1. The setting studied in this paper is interesting and realistically challenging, as it is often unclear how to use such auxiliary feedback for better online decision-making.\n2. The paper is well organized. The theoretical results also appear sound. \n3. The authors have provided a comprehensive literature review on existing research related to this work, and clearly explained the connection between their work and prior works in control variate theory.\n\nWEAKNESSES:\nIt would be good if the authors could provide further discussions and clarifications on the following points:\n1. The description of the algorithm (OFUL-AF) could be made clearer. Currently, it appears as a rephrasing of each step without clear connections to prior derivations. It would be helpful to explain how each step relates to the preceding derivations.\n2. The procedure of selection of the number of auxiliary feedback $q$ in practice is not entirely clear to me. It'd be helpful to have more details regarding this point in addition to Remark 1. \n3. The regret bound in Theorem 2 contains hidden constant terms. Could you elaborate more on the magnitude of these constant terms and their impact on the overall result?\n4. The authors acknowledge that the actual form of the auxiliary feedback functions is typically unknown in practice, which I consider to be an important realistic challenge that needs to be dealt with. While Section 4 explores the effect of using estimated functions, I wonder how one can obtain an unbiased estimator for the auxiliary feedback functions in the first place. This is a challenging aspect that merits further explanation.\n5. How should I comprehend the terms $a(e)$ and $\\rho_e$ from Theorem 3? What do they each represent and do they constitute certain kind of tradeoff within the regret bound? \n6. It'd be helpful to also see how the proposed model and method can be applied to certain real-world data. This is related to my point (4) above as I'm not entirely sure how one'd make sense of the auxiliary feedback function in practice.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper leverages the method of control variates to obtain reward estimates with smaller variance for contextual bandit algorithms, since smaller variance in reward estimation means tighter confidence bound estimation and therefore smaller regret. Estimation methods for both unknown and known auxiliar feedback functions are provided, where both theoretical and empirical results were provided to demonstrate the effectiveness of the proposed solution.\n\nSTRENGTHS:\nLeveraging available auxiliar feedback to improve reward estimation is an important and also meaningful approach for improving bandit algorithms. This paper provides a viable solution to realize the goal. \n\nThe provided solution is an extension of Verma and Hanawal’s NeurIPS’2021 work in multi-armed bandits, and the authors further extended it to contextual bandit problems, especially with the non-linear reward functions.\n\nWEAKNESSES:\nThe means of using control variates to reduce the reward estimation variance is not a completely new idea, as the authors pointed out in the related work discussions. And the estimation techniques employed in this paper were also borrowed from prior works. For example, the results under known auxiliary feedback function is a straightforward extension of Verma and Hanawal’s NeurIPS’2021 work on top of OFUL’s analysis, and the results for the unknown auxiliary feedback function is also not super challenging (e.g., using existing estimation technique and assuming the reward variance is known). This to certain degree limits the novelty of this work. \n\nThe description for the unbiased linear estimator $\\beta_e$ is unfortunately unclear and could be actually problematic. Based on the problem setup, my understanding is that every arm pull reveals all $q$ dimensions of the auxiliary feedback, in addition to the reward feedback. And the different sampling/partitioning methods, e.g., IS or MF, decide how to allocate those samples for estimating the reward function and auxiliary feedback function. Hence, the total number of samples for estimation at time $t$ is $t$, but each sample has $q+1$ dimensions. But in line 302, it states to maintain $r=2$, it needs to “getting one extra sample of auxiliary feedback in each round”. This seems to suggest the algorithm can pull another arm and only require the auxiliary feedback for free. If this is the case for algorithm design, it is an unfair advantage to the algorithm, comparing to the baseline bandit algorithms, as it can collect more information about the reward. Or in other words, why not further increase the ratio $r$ to better exploit this advantage? The EH variant of this algorithm further confirmed my understanding: it is assumed more observations about the auxiliary feedback are available.  \n\nThe experiment settings were also overly simplified, which do not strongly support the advantages or demonstrate the limitations of the proposed solution. For example, there is only one dimension of auxiliary feedback. Given the algorithm’s theoretical performance depends on the dimension of auxiliary feedback, it is important to vary its dimension to investigate its practical impact. \n\nAnother factor should be mentioned is that the paper only addresses the finite arm setting (especially in the experiments), though in contextual bandit problems infinite arms with potentially adversarial context arrival is believed to be a more general setting. Otherwise, simple algorithms can already achieve satisfactory regret: for example, a greedy algorithm can obtain sublinear regret with sufficient context diversity and logarithmic regret is also achievable under stochastic context distributions. It would be meaningful to discuss how the developed algorithm can be extended to this more general and also more challenging environment.\n\nQUESTIONS:\nSince we are using linear regression for $\\beta$ estimation, why is $W^\\top_t W_t$ guaranteed to be invertible? Even when we have more samples than the number of auxiliary dimensions, i.e., $t>q+2$, the observations of auxiliary feedback might not span the entire $q$ dimensional space, depending on the distribution of selected arms.   \n\nIs it true that the algorithm is supposed to have free access to auxiliary feedback from any arm? If so, what presents the algorithm from extensively pulling all the arms to estimate their auxiliary feedback function, so as to get the most accurate estimation of those functions first?\n\nThe algorithm assumes the reward noise is known and use this quantity to control when to use which estimator. But in practice, we do not know the actual value of reward variance, and how could we use the proposed algorithm?\n\nLIMITATIONS:\nI do not find any concerns regarding the negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the parametrized bandit problem in which the learner observes auxiliary feedback together with the reward, and also correlated with the reward. It is motivated from the control variate approach in causal inference, the main difference is that in this paper, it extends the control variable theory to a setting where the \"control variable\" is parametrized by a function. It proposes a new bandit algorithm which replaces the original observed reward, with a version built upon the known/estimated control variate and studies the expected instantaneous regret compared with the original bandit algorithm.\n\nSTRENGTHS:\n- This paper studies a new bandit framework which differs from most of the prior work (such as side information, side observation, etc). In this new framework, the learner also observes the auxiliary feedback beyond the original reward. This setting is pretty relevant to lots of real-world scenarios, such as in food delivery platform, user rating might be revealed together with the delivery time, in recommendation platform, optimizing user like rate might be revealed together with the watch time, etc. \n\n- The paper is very well-written and easy to follow. It is built upon the classical control variate theory, and extends it to the known function (of the control variate), then further down to estimated function setting. The method is solid, and the expected instantaneous regret is provided to validate the soundness of the proposed approach.\n\n- I appreciate the additional efforts in the synthetic datasets to verify the empirical performance of the proposed method under various environment, such as the linear, linear contextual, non-linear contextual bandits, and study how the estimation of the control variate function as well as the correlation of the auxiliary feedback with the reward affect the final performance. These results facilitate the understanding.\n\nWEAKNESSES:\n- I am a little bit confused about the relationship of the variance reduction in terms of the number of auxiliary feedback being used. Under estimated $\\beta$ and from Theorem 1, it seems the optimal number of auxiliary feedback being used is 1, which seems a little bit counter-intuitive, could the authors comment more on this aspect?\n\n- In Section 4, under estimated auxiliary functions, the sampling strategy for estimating the auxiliary function seems very computationally inefficient, and this leaves much smaller sample size for estimating the original f function compared with classical bandit algorithms, especially when the number of auxiliary functions is large. Remark 2 also does not make that much of sense, when $r_i$ goes to infinity where we use most of the sample in auxiliary function estimate. \n\n- The IS and MF sampling strategy are only listed in the method discussions, and for the experiments when $q=2$, it is hard to compare the pros and cons of them as they are equivalently being the same. Could the authors add more results for larger $q$ to showcase the effectiveness of the two sampling methods. A larger $q$ might also be helpful in understanding the effectiveness of having more control variates. \n\n- I am not sure if there is any public dataset available to showcase the effectiveness of the method in more real-world scenarios. The current ablation experiments facilitate the understanding of the method, but adding real-world experiments would be much more convincing.\n\nQUESTIONS:\n- For Figure 2 (f), is it possible to add the performance for original Lin-UCB performance?\n- It would be good to have an ablation study w.r.t the number of auxiliary functions.\n- others listed in the Weakness section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper focus on the problem of parameterized bandits when extra auxiiliary feedback is available, which can be utilized to construct an unbiased reward estimator, which potentially shares smaller variance and thus algorithms based on such estimator can thus incur smaller regret. Experiments validates the estimator and corresponding algorithm.

STRENGTHS:
- The paper is clearly-written, in which the problem statement, algorithms, theories and experiments are easy to follow.
- Control variate theory is applied in this paper, which is suitable for i.i.d. random process in the variance reduction, which is one of the central topic in bandits, and is of independent interests beyond the topic.
- The proposed algorithm is evaluated both empirically and theoretically.

WEAKNESSES:
- The improvement is expectable, as auxiliary feedback (AF) requires more information and computation. So this can be seen as a trade-off between extra information beyond rewards, extra computation and a constant order improvement in regret.
- The hybrid reward only reduces variance when the AF is correlated with reward with same covariance $\sigma_{y,w}$. I'm worried this could not be true for most real-world applications. 
- Though extra AF is given, the experiments does not design how OFUL or LinUCB can utilize such AFs. One can figure another simplest way to use these extra information, for example, learn a model $f:\mathbf{R}^d \rightarrow \mathbf{R}$, which means to learn the function that maps the AF vector to the reward $y_t$, and thus OFUL and LinUCB can gain extra feedback $f(AF)$ to better estimate the reward. If the baseline just discard the extra AF, its' not fair.

QUESTIONS:
See weaknesses above.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05