Partial Information as Full: Reward Imputation with Sketching in Bandits

1 INTRODUCTION . Contextual bandits have been widely used in real-world sequential decision-making problems ( Li et al. , 2010 ; Lan & Baraniuk , 2016 ; Yom-Tov et al. , 2017 ; Yang et al. , 2021 ) , where the agent updates the decision-making policy fully online ( i.e. , at each step ) according to the context and corresponding reward feedback so as to maximize the cumulative reward . In this paper , we consider a more complex setting—contextual batched bandits ( CBB ) , where the decision process is partitioned into N episodes , the agent interacts with the environment for B steps in one episode , collects the reward feedbacks and contexts at the end of the episode , and then updates the policy using the collected data for the next episode . CBB is more practical in some real-world applications , since updating the policy once receiving the reward feedback is rather unrealistic due to its high computational cost and decision instability . In bandit settings , it is inevitable that the environment only reveals the rewards of the executed actions to the agent as the feedbacks , while hiding the rewards of non-executed actions . We refer to this category of limited feedback as the partial-information feedback ( also called “ bandit feedback ” ) . Existing batched bandit approaches in the CBB setting discard the information contained in the potential rewards of the non-executed actions , address the problem of partial-information feedback using an exploitation-exploration tradeoff on the context space and reward space ( Han et al. , 2020 ; Zhang et al. , 2020 ) .But in the CBB setting , the agent usually estimates and maintains reward models for the action-selection policy , and the potential rewards of the non-executed actions have been somehow captured by the policy . This additional reward structure information is estimated and available in each episode , however , are not utilized by existing batched bandit approaches . In contextual bandit settings where the policy is updated fully online , several bias-correction approaches have been introduced to address the partial-information feedback . Dimakopoulou et al . ( 2019 ) presented linear contextual bandits integrating the balancing approach from causal inference , which reweight the contexts and rewards by the inverse propensity scores . Chou et al . ( 2015 ) designed pseudo-reward algorithms for contextual bandits , which use a direct method to estimate the unobserved rewards for the upper confidence bound ( UCB ) strategy . Kim & Paik ( 2019 ) focused on the feedback bias-correction for LASSO bandit with high-dimensional contexts , and applied the doubly-robust approach to the reward modification using average contexts . Although these approaches have been demonstrated to be effective in contextual bandit settings , little efforts have been spent to address the under-utilization of partial-information feedback in the CBB setting . Theoretical and experimental analyses in Section 2 indicate that better performance of CBB is achievable if the rewards of the non-executed actions can be received . Motivated by these observations , we propose a novel reward imputation approach for the non-executed actions , which mimics the reward generation mechanisms of environments . We conclude our contributions as follows . • To fully utilize feedback information in CBB , we formulate the reward imputation as a problem of imputation regularized ridge regression , where the policy can be updated efficiently using sketching . • We prove that our reward imputation approach obtains a relative-error bound for sketching approximation , achieves an instantaneous regret with a controllable bias and a smaller variance than that without reward imputation , has a lower bound of the sketch size independently of the overall number of steps , enjoys a sublinear regret bound against the optimal policy , and reduces the time complexity from O ( Bd2 ) to O ( cd2 ) for each action in one episode , where B denotes the batch size , c the sketch size , and d the dimensionality of inputs , satisfying d < c < B . • We present two practical variants of our reward imputation approach , including the rate-scheduled version in which the imputation rate is set without tuning , and the version for nonlinear rewards . • We carried out extensive experiments on the synthetic data , public benchmark , and the data collected from a real commercial product to demonstrate our performance , empirically analyzed the influence of different parameters , and verified the correctness of the theoretical results . Related Work . Recently , batched bandit has become an active research topic in statistics and learning theory including 2-armed bandit ( Perchet et al. , 2016 ) , multi-armed bandit ( Gao et al. , 2019 ; Zhang et al. , 2020 ; Wang & Cheng , 2020 ) , and contextual bandit ( Han et al. , 2020 ; Ren & Zhou , 2020 ; Gu et al. , 2021 ) . Han et al . ( 2020 ) defined linear contextual bandits , and designed UCB-type algorithms for both stochastic and adversarial contexts , where true rewards of different actions have the same parameters . Zhang et al . ( 2020 ) provided methods for inference on data collected in batches using bandits , and introduced a batched least squares estimator for both multi-arm and contextual bandits . Recently , Esfandiari et al . ( 2021 ) proved refined regret upper bounds of batched bandits in stochastic and adversarial settings . There are several recent works that consider similar settings to CBB , e.g. , episodic Markov decision process ( Jin et al. , 2018 ) , LASSO bandits ( Wang & Cheng , 2020 ) . Sketching is another related technology that compresses a large matrix to a much smaller one by multiplying a ( usually ) random matrix with certain properties ( Woodruff , 2014 ) , which has been used in online convex optimization ( Calandriello et al. , 2017 ; Zhang & Liao , 2019 ) . 2 PROBLEM FORMULATION AND ANALYSIS . First , we introduce some notations . Let [ x ] = { 1 , 2 , . . . , x } , S ⊆ Rd be the context space , A = { Aj } j∈ [ M ] the action space containing M actions , [ A ; B ] = [ Aᵀ , Bᵀ ] ᵀ , ‖A‖F , ‖A‖1 ‖A‖2 denote the Frobenius norm , 1-norm , and spectral norm of a matrix A , respectively , ‖a‖1 and ‖a‖2 be the ` 1-norm and the ` 2-norm of a vector a , σmin ( A ) and σmax ( A ) denote the minimum and maximum of the of singular values of A . In this paper , we focus on the setting of Contextual Batched Bandits ( CBB ) ( see Algorithm 1 ) , where the decision process is partitioned into N episodes , and in each episode , CBB consists of two phases : 1 ) the policy updating approximates the optimal policy based on the received contexts and rewards ; 2 ) the online decision selects actions for execution following the updated and fixed policy p for B steps ( also called the batch size is B ) , and stores the contextaction pairs and the observed rewards of the executed actions into a data buffer D. The reward R in CBB is a partial-information feedback where rewards are unobserved for the non-executed actions . Different from existing batch bandit setting ( Han et al. , 2020 ; Esfandiari et al. , 2021 ) , where the true reward feedbacks for all actions are controlled by the same parameter vector while the contexts received differ by actions at each step , we assume that in the CBB setting , the mechanism of true reward feedback differs by actions and the received context is shared by actions . Formally , for any context si ∈ S ⊆ Rd and action A ∈ A , we assume that the expectation of the true reward Rtruei , A Algorithm 1 Contextual Batched Bandit ( CBB ) INPUT : Batch size B , number of episodes N , action space A = { Aj } j∈ [ M ] , context space S ⊆ Rd 1 : Initialize policy p0 ← 1/M , sample data buffer D1 = { ( s0 , b , AI0 , b , R0 , b ) } b∈ [ B ] using initial policy p0 2 : for n = 1 to N do 3 : Update the policy pn on Dn { Policy Updating } 4 : for b = 1 to B do 5 : Observe context sn , b 6 : Choose AIn , b ∈ A following the updated policy pn ( sn , b ) { Online Decision } 7 : end for 8 : Dn+1 ← { ( sn , b , AIn , b , Rn , b } b∈ [ B ] , where Rn , b denotes the reward of action AIn , b on context sn , b 9 : end for is determined by an unknown action-specific reward parameter vector θ∗A ∈ Rd : E [ Rtruei , A | si ] = 〈θ∗A , si〉 ( the linear reward will be extended to the nonlinear case in Section 5 ) . This setting for reward feedback matches many real-world applications , e.g. , each action corresponds to a different category of candidate coupons in coupon recommendation , and the reward feedback mechanism of each category differs due to the different discount pricing strategies . Next , we provide deeper understandings of the influence of unobserved feedbacks on the performance of policy updating in the CBB setting . We first conducted an empirical comparison by applying the batch UCB policy ( Han et al. , 2020 ) to environments under different proportions of received reward feedbacks . In particular , the agent under full-information feedback can receive all the rewards of the executed and non-executed actions , called Full-Information CBB ( FICBB ) setting . From Figure 1 , we can observe that the partial-information feedbacks could be damaging in terms of hurting the policy updating , and batched bandit could benefit from more reward feedbacks , where the performance of 80 % feedback is very close to that of FI-CBB . Then , we demonstrate the difference of instantaneous regrets between the CBB and FI-CBB settings as in Theorem 1 . The detailed description and proof of Theorem 1 can be found in Appendix A. Theorem 1 . For any action A ∈ A and context si ∈ S , let θnA be the reward parameter vector estimated by the batched UCB policy in the nth episode . The upper bound of instantaneous regret ( defined by |〈θnA , si〉 − 〈θ∗A , si〉| ) in the FI-CBB setting is tighter than that in CBB setting ( i.e. , using the partial-information feedback ) . From Theorem 1 , we can conclude that the price paid for having access only to the partial-information feedbacks is the deterioration in the regret bound . Ideally , the policy would be updated using the fullinformation feedback . In CBB , however , the fullinformation feedback is unaccessible . Fortunately , in CBB , different reward parameter vectors need to be maintained and estimated separately for each action , and the potential reward structures of the non-executed actions have been somehow captured . So why don ’ t we use these maintained reward parameters to estimate the unknown rewards for the non-executed actions ? Next , we propose an efficient reward imputation approach that uses this additional reward structure information for improving the performance of the bandit policy . 3 EFFICIENT REWARD IMPUTATION FOR POLICY UPDATING . In this section , we present an efficient reward imputation approach tailored for policy updating in the CBB setting . Formulation of Reward Imputation . Since the true reward parameters differ by actions in the CBB setting , we need to maintain a different estimated reward parameter vector for each action . As shown in Figure 2 , in contrast to CBB that ignores the contexts and rewards of the non-executed steps of Aj , our reward imputation approach completes the missing values using the imputed contexts and rewards , approximating the full-information CBB setting . Specifically , in the ( n+1 ) -th episode , for each action Aj ∈ A , j ∈ [ M ] , we store context vectors and rewards corresponding to the steps in which the action Aj is executed , into a context matrix SnAj ∈ R Nnj ×d and a reward vector RnAj ∈ RN n j , respectively , whereNnj denotes the number of steps ( in episode n+1 ) in which the actionAj is executed . Additionally , for anyAj ∈ A , j ∈ [ M ] , we store context vectors corresponding to the nonexecuted steps of action Aj ( denote the number of non-executed steps by N̂nj , i.e. , N̂ n j = B−Nnj ) , into an imputed context matrix ŜnAj ∈ R N̂nj ×d , and compute the imputed reward vector as follows : R̂nAj = { rn,1 ( Aj ) , rn,2 ( Aj ) , . . . , rn , N̂nj ( Aj ) } ∈ R N̂nj , j ∈ [ M ] , where rn , b ( Aj ) : = 〈θ̄nAj , sn , b〉 denotes the imputed reward for each step b ∈ [ N̂ n j ] , and sn , b is the b-th row of ŜnAj . Then , we obtain several block matrices by concatenating the context and reward matrices from the previous episodes : LnAj = [ S 0 Aj ; · · · ; SnAj ] ∈ R Lnj ×d , T nAj = [ R 0 Aj ; · · · ; RnAj ] ∈ RL n j , Lnj = ∑n k=0N k j , L̂ n Aj = [ Ŝ0Aj ; · · · ; Ŝ n Aj ] ∈ RL̂ n j ×d , T̂ nAj = [ R̂ 0 Aj ; · · · ; R̂nAj ] ∈ R L̂nj , L̂nj =∑n k=0 N̂ k j . In the ( n + 1 ) -th episode , the estimated parameter vector θ̄ n+1 Aj of the imputed reward for action Aj can be updated by solving the following imputation regularized ridge regression : θ̄n+1Aj = arg min θ∈Rd ∥∥∥LnAjθ − T nAj∥∥∥2 2︸ ︷︷ ︸ Observed Term + γ ∥∥∥L̂nAjθ − T̂ nAj∥∥∥2 2︸ ︷︷ ︸ Imputation Term +λ‖θ‖22 , n = 0 , 1 , . . . , N − 1 , ( 1 ) where γ ∈ [ 0 , 1 ] is the imputation rate which controls the degree of reward imputation and measures a trade-off between bias and variance ( Remark 1 & 2 ) , λ > 0 is the regularization parameter , yielding θ̄n+1Aj = ( Ψn+1Aj ) −1 ( bn+1Aj + γb̂ n+1 Aj ) , ( 2 ) that can be obtained by the closed least squares solution , where Ψn+1Aj : = λId + Φ n+1 Aj + γΦ̂n+1Aj , Φn+1Aj = Φ n Aj + S nᵀ Aj SnAj , b n+1 Aj = bnAj + S nᵀ Aj RnAj , ( 3 ) Φ̂n+1Aj = ηΦ̂ n Aj + Ŝ nᵀ Aj ŜnAj , b̂ n+1 Aj = ηb̂nAj + Ŝ nᵀ Aj R̂nAj , ( 4 ) and η ∈ ( 0 , 1 ) is the discount parameter which controls how fast the previous imputed rewards are forgotten , and can help guaranteeing the regret bound in Theorem 2 . Efficient Reward Imputation using Sketching . As shown in the first 4 columns in Table 1 , the overall time complexity of the imputation for each action is O ( Bd2 ) in each episode , where B represents the batch size , and d the dimensionality of the input . Thus , for all the M actions in one episode , reward imputation increases the time complexity from O ( Bd2 ) of the approach without imputation to O ( MBd2 ) . To address this issue , we design an efficient reward imputation approach using sketching , which reduces the time complexity of each action in one episode from O ( Bd2 ) to O ( cd2 ) , where c denotes the sketch size satisfying d < c < B and cd > B . Specifically , in the ( n+ 1 ) -th episode , the imputation regularized ridge regression Eq . ( 1 ) can be approximated by a sketched ridge regression as θ̃n+1Aj = arg min θ∈Rd ∥∥∥ΠnAj ( LnAjθ − T nAj ) ∥∥∥2 2 + γ ∥∥∥Π̂nAj ( L̂nAjθ − T̂ nAj ) ∥∥∥2 2 + λ‖θ‖22 , ( 5 ) where θ̃n+1A denotes the estimated parameter vector of the imputed reward using sketching for action A ∈ A , CnAj ∈ R c×Nnj and ĈnAj ∈ R c×N̂nj are the sketch submatrices for the observed term and the imputation term , respectively , and the sketch matrices for the two terms can be represented as ΠnAj = [ C0Aj , C 1 Aj , · · · , C n Aj ] ∈ Rc×L n j , Π̂nAj = [ Ĉ0Aj , Ĉ 1 Aj , · · · , Ĉ n Aj ] ∈ Rc×L̂ n j . We denote the sketches of the context matrix and the reward vector by ΓnAj : = C n Aj SnAj ∈ R c×d and ΛnAj : = C n Aj RnAj ∈ R c , the sketches of the imputed context matrix and the imputed reward vector by Γ̂nAj : = Ĉ n Aj ŜnAj ∈ R c×d and Λ̂nAj : = Ĉ n Aj R̂nAj ∈ R c , and obtain the solution of Eq . ( 5 ) : θ̃n+1Aj = ( W n+1Aj ) −1 ( pn+1Aj + γp̂ n+1 Aj ) , ( 6 ) where η ∈ ( 0 , 1 ) denotes the discount parameter , W n+1Aj : = λId +G n+1 Aj + γĜn+1Aj , and Gn+1Aj = G n Aj + Γ nᵀ Aj ΓnAj , p n+1 Aj = pnAj + Γ nᵀ Aj ΛnAj , ( 7 ) Ĝn+1Aj = ηĜ n Aj + Γ̂ nᵀ Aj Γ̂nAj , p̂ n+1 Aj = ηp̂nAj + Γ̂ nᵀ Aj Λ̂nAj . ( 8 ) Using the parameter θ̃n+1Aj , we obtain the sketched version of imputed reward as r̃n , b ( Aj ) : = 〈θ̃nAj , sn , b〉 at step b ∈ [ N̂ n j ] . Finally , we specify that the sketch submatrices { CnA } A∈A , n∈ [ N ] and { ĈnAj } A∈A , n∈ [ N ] are the block construction of Sparser Johnson-Lindenstrauss Transform ( SJLT ) ( Kane & Nelson , 2014 ) , where the sketch size c is divisible by the number of blocks D1 . As shown in the last 4 columns in Table 1 , sketching reduces the time complexity fromO ( MBd2 ) toO ( Mcd2 ) for reward imputation of all M actions in one episode , where c < B . When Mc ≈ B , the overall time complexity of our reward imputation using sketching is even comparable to that without reward imputation which has a O ( Bd2 ) time complexity . Updated Policy using Imputed Rewards . Inspired by the UCB strategy ( Li et al. , 2010 ) , the updated policy for online decision of the ( n+ 1 ) -th episode can be formulated using the imputed rewards ( parameterized by θ̄n+1A in Eq . ( 2 ) ) or the sketched version of imputed rewards ( parameterized by θ̃n+1A in Eq. ( 6 ) ) . Specifically , for a new context s , origin policy p̄n+1 selects the action following A← arg max A∈A 〈θ̄n+1A , s〉+ ω [ sᵀ ( Ψ n+1 A ) −1s ] 1 2 , sketched policy p̃n+1 selects the action following A ← arg max A∈A 〈θ̃n+1A , s〉 + α [ sᵀ ( W n+1 A ) −1s ] 1 2 , where ω ≥ 0 and α ≥ 0 are the regularization parameters in policy and their theoretical values are given in Theorem 4 . We summarize the reward imputation using sketching and the sketched policy into Algorithm 2 , called SPUIR . Similarly , we call the updating of the original policy that uses reward imputation without sketching , the Policy Updating with Imputed Rewards ( PUIR ) . 1Since we set the number of blocks of SJLT as D < d , we omit D in the complexity analysis . Algorithm 2 Sketched Policy Updating with Imputed Rewards ( SPUIR ) in the ( n+ 1 ) -th episode INPUT : Policy p̃n , Dn+1 , A = { Aj } j∈ [ M ] , α ≥ 0 , η ∈ ( 0 , 1 ) , γ ∈ [ 0 , 1 ] , λ > 0 , W 0Aj = λId , G 0 Aj = Ĝ0Aj = Od , p 0 Aj = p̂0Aj = 0 , θ̃ 0 Aj = 0 , j ∈ [ M ] , batch size B , sketch size c , number of block D OUTPUT : Updated policy p̃n+1 1 : For all j ∈ [ M ] , store context vectors and rewards corresponding to the steps in which the action Aj is executed , into ΓnAj ∈ R Nnj ×d and ΛnAj ∈ R Nnj 2 : For all j ∈ [ M ] , store context vectors corresponding to the steps in which the action Aj is not executed into Γ̂nAj ∈ R N̂nj ×d , where N̂nj ← B −Nnj 3 : r̃n , b ( Aj ) ← 〈θ̃nAj , sn , b〉 , for all Aj ∈ A and b ∈ [ N̂ n j ] , where sn , b is the b-th row of Γ̂ n Aj 4 : Compute imputed reward vector R̂nAj ← { r̃n,1 ( Aj ) , . . . , r̃n , N̂nj ( Aj ) } ∈ R N̂nj for any j ∈ [ M ] 5 : for all action Aj ∈ A do 6 : Gn+1Aj ← G n Aj + ΓnᵀAjΓ n Aj , pn+1Aj ← p n Aj + ΓnᵀAjΛ n Aj { Eq . ( 7 ) } 7 : Ĝn+1Aj ← ηĜ n Aj + Γ̂nᵀAj Γ̂ n Aj , p̂n+1Aj ← ηp̂ n Aj + Γ̂nᵀAj Λ̂ n Aj { Eq . ( 8 ) } 8 : W n+1Aj ← λId +G n+1 Aj + γĜn+1Aj , θ̃ n+1 Aj ← ( W n+1Aj ) −1 ( pn+1Aj + γp̂ n+1 Aj ) { Eq . ( 6 ) } 9 : end for 10 : p̃n+1 ( s ) selects action A← argmaxA∈A〈θ̃ n+1 A , s〉+ α [ s ᵀ ( W n+1A ) −1 s ] 1 2 for a new context s 11 : return { θ̃n+1A } A∈A , { ( W n+1A ) −1 } A∈A

Summary: The paper considers the behavior of a series of algorithms for learning in a batched bandit setting, where parameters are arm/action specific, rather than shared across arms/actions.   These algorithms are based on UCB like techniques with the addition of a regularization term that aims to use information from imputed, unobserved rewards. Such techniques are present in the linear regression and dimensionality reduction literature where one aims to use either prior information or  data distribution information to aid prediction. However, to my knowledge, these formalisms are quite  new to online settings like bandits, thus making the presented approach appealing.