Title: Reward Imputation with Sketching for Contextual Batched Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: batched bandit

sketching

reward imputation

regret bound

ridge regression
Abstract: Contextual batched bandit (CBB) is a setting where a batch of rewards is observed from the environment at the end of each episode, but the rewards of the non-executed actions are unobserved, resulting in partial-information feedback. Existing approaches for CBB often ignore the rewards of the non-executed actions, leading to underutilization of feedback information. In this paper, we propose an efficient approach called Sketched Policy Updating with Imputed Rewards (SPUIR) that completes the unobserved rewards using sketching, which approximates the full-information feedbacks. We formulate reward imputation as an imputation regularized ridge regression problem that captures the feedback mechanisms of both executed and non-executed actions. To reduce time complexity, we solve the regression problem using randomized sketching. We prove that our approach achieves an instantaneous regret with controllable bias and smaller variance than approaches without reward imputation. Furthermore, our approach enjoys a sublinear regret bound against the optimal policy. We also present two extensions, a rate-scheduled version and a version for nonlinear rewards, making our approach more practical. Experimental results show that SPUIR outperforms state-of-the-art baselines on synthetic, public benchmark, and real-world datasets.

Reward Imputation with Sketching for Contextual Batched Bandits

Abstract

Contextual batched bandit (CBB) is a setting where a batch of rewards is observed from the environment at the end of each episode, but the rewards of the non-executed actions are unobserved, resulting in partial-information feedback. Existing approaches for CBB often ignore the rewards of the non-executed actions, leading to underutilization of feedback information. In this paper, we propose an efficient approach called Sketched Policy Updating with Imputed Rewards (SPUIR) that completes the unobserved rewards using sketching, which approximates the full-information feedbacks. We formulate reward imputation as an imputation regularized ridge regression problem that captures the feedback mechanisms of both executed and non-executed actions. To reduce time complexity, we solve the regression problem using randomized sketching. We prove that our approach achieves an instantaneous regret with controllable bias and smaller variance than approaches without reward imputation. Furthermore, our approach enjoys a sublinear regret bound against the optimal policy. We also present two extensions, a rate-scheduled version and a version for nonlinear rewards, making our approach more practical. Experimental results show that SPUIR outperforms state-of-the-art baselines on synthetic, public benchmark, and real-world datasets.

1 Introduction

Contextual bandits have gained significant popularity in solving sequential decision-making problems (Li et al., 2010; Lan and Baraniuk, 2016; Yom-Tov et al., 2017; Yang et al., 2021), where the agent continuously updates its decision-making policy fully online (i.e., at each step), considering the context and the received reward feedback to maximize cumulative rewards. In this paper, we address a more general setting called contextual batched bandits (CBB). In CBB, the decision process is divided into N episodes, and within each episode, the agent interacts with the environment for a fixed number of B steps. At the end of each episode, the agent collects reward feedbacks and contexts. Subsequently, the policy is updated using the collected data to guide the decision-making process in the subsequent episode. CBB offers a practical framework for real-world streaming applications (e.g., streaming recommendation (Zhang et al., 2021, 2022)). In the context of CBB settings, the batch size B , can be adjusted by the agent to achieve improved regret guarantees and meet the data throughput requirements based on the available computing resources (Zhou, 2023).

In bandit settings, it is common for the environment to only provide feedback on the rewards of executed actions to the agent, while concealing the rewards of non-executed actions. This type of limited feedback is referred to as partial-information feedback (also called 'bandit feedback'). In CBB setting, existing approaches tend to overlook the potential rewards associated with non-executed actions. Instead, they address the challenge of partial-information feedback through an explorationexploitation tradeoff in both the context space and reward space (Han et al., 2020; Zhang et al., 2020). However, CBB agents typically estimate and maintain reward models for the action-selection policy, thereby capturing some information about the potential rewards of non-executed actions. This additional reward structure information is available for policy updating in each episode but remains untapped by existing batched bandit approaches.

In the context of contextual bandit settings where the policy is updated online, several bias-correction approaches have been introduced to tackle the issue of partial-information feedback. Dimakopoulou et al. (2019) presented linear contextual bandits integrating the balancing approach from causal inference, which reweight the contexts and rewards by the inverse propensity scores. Chou et al. (2015) designed pseudo-reward algorithms for contextual bandits using upper confidence bound (UCB) strategy, which use a direct method to estimate the unobserved rewards. Kim and Paik (2019) focused on the correction of feedback bias for LASSO bandit with high-dimensional contexts, and applied the doubly-robust approach to the reward modification using average contexts. While these approaches have demonstrated effectiveness in contextual bandit settings, little attention has been given to addressing the under-utilization of partial-information feedback in CBB setting.

Theoretical and experimental analyses in Section 2 indicate that better performance of CBB is achievable if the rewards of the non-executed actions can be received. Motivated by these observations, we propose a novel reward imputation approach for the non-executed actions, which mimics the reward generation mechanisms of environments. We conclude our contributions as follows.

(1) To fully utilize feedback information in CBB, we formulate the reward imputation as a problem of imputation regularized ridge regression, where the policy can be updated efficiently using sketching.

(2) We prove that our reward imputation approach obtains a relative-error bound for sketching approximation, achieves an instantaneous regret with a controllable bias and a smaller variance than that without reward imputation, has a lower bound of the sketch size independently of the overall number of steps, enjoys a sublinear regret bound against the optimal policy, and reduces the time complexity from O ( Bd 2 ) to O ( cd 2 ) for each action in one episode, where B denotes the batch size, c the sketch size, and d the dimension of the context space, satisfying d < c < B .

(3) We present two practical variants of our reward imputation approach, including the rate-scheduled version that sets the imputation rate without tuning, and the version for nonlinear rewards.

(4) We carried out extensive experiments on a synthetic dataset, the publicly available Criteo dataset, and a dataset from a commercial app to demonstrate our performance, empirically analyzed the influence of different parameters, and verified the correctness of the theoretical results.

Related Work. Recently, batched bandit has become an active research topic in statistics and learning theory including 2 -armed bandit (Perchet et al., 2016), multi-armed bandit (Gao et al., 2019; Zhang et al., 2020; Wang and Cheng, 2020), and contextual bandit (Han et al., 2020; Ren and Zhou, 2020; Gu et al., 2021). Han et al. (2020) defined linear contextual bandits, and designed UCB-type algorithms for both stochastic and adversarial contexts, where true rewards of different actions have the same parameters. Zhang et al. (2020) provided methods for inference on data collected in batches using bandits, and introduced a batched least squares estimator for both multi-arm and contextual bandits. Recently, Esfandiari et al. (2021) proved refined regret upper bounds of batched bandits in stochastic and adversarial settings. There are several recent works that consider similar settings to CBB, e.g., episodic Markov decision process (Jin et al., 2018), LASSO bandits (Wang and Cheng, 2020). Sketching is another related technology that compresses a large matrix to a much smaller one by multiplying a (usually) random matrix while retaining certain properties (Woodruff, 2014), which has been used in online convex optimization (Calandriello et al., 2017; Zhang and Liao, 2019).

2 Problem Formulation and Analysis

Let [ x ] = { 1 , 2 , . . . , x } , S ⊆ R d be the context space whose dimension is d , A = { A j } j ∈ [ M ] the action space containing M actions, [ A ; B ] = [ A ᵀ , B ᵀ ] ᵀ , ‖ A ‖ F , ‖ A ‖ 1 ‖ A ‖ 2 denote the Frobenius

Protocol 1 Contextual Batched Bandit (CBB)

norm, 1 -norm, and spectral norm of a matrix A , respectively, ‖ a ‖ 1 and ‖ a ‖ 2 be the glyph[lscript] 1 -norm and the glyph[lscript] 2 -norm of a vector a , σ min ( A ) and σ max ( A ) denote the minimum and maximum of the of singular values of A . In this paper, we focus on the setting of Contextual Batched Bandits (CBB) in Protocol 1, where the decision process is partitioned into N episodes, and in each episode, CBB consists of two phases: (1) the policy updating approximates the optimal policy based on the received contexts and rewards; (2) the online decision chooses actions for execution following the updated and fixed policy p for B steps ( B is also called the batch size ), and stores the context-action pairs and the observed rewards of the executed actions into a data buffer D . The reward R in CBB is a partial-information feedback where rewards are unobserved for the non-executed actions.

In contrast to the existing batched bandit setting (Han et al., 2020; Esfandiari et al., 2021), where the true reward feedbacks for all actions are controlled by the same parameter vector while the received contexts differ across actions at each step, we make the assumption that in CBB setting, the mechanism of true reward feedback varies across actions, while the received context is shared among actions. Formally, for any context s i ∈ S ⊆ R d and action A ∈ A , we assume that the expectation of the true reward R true i,A is determined by an unknown action-specific reward parameter vector θ ∗ A ∈ R d : E [ R true i,A | s i ] = 〈 θ ∗ A , s i 〉 (the linear reward will be extended to the nonlinear case in Section 5). This setting for reward feedback matches many real-world applications, e.g., each action corresponds to a different category of candidate coupons in coupon recommendation, and the reward feedback mechanism of each category differs due to the different discount pricing strategies.

Next, we delve deeper into understanding the impact of unobserved feedbacks on the performance of policy updating in CBB setting. We first conducted an empirical comparison by applying the batch UCB policy (SBUCB) (Han et al., 2020) to environments under different proportions of received reward feedbacks. In particular, the agent under full-information feedback can receive all the rewards of the executed and non-executed actions, called Full-Information CBB (FI-CBB) setting. From Figure 1, we can observe that the partial-information feedbacks are damaging in terms of hurting the policy updating, and batched bandit policy can benefit from more reward feedbacks, where the performance of 80% feedback is very close to that of FI-CBB.

Then, we prove the difference of instantaneous re- grets between the CBB and FI-CBB settings in

Theorem 1 (proof can be found in Appendix A).

Theorem 1. For any action A ∈ A and context s i ∈ S , let θ n A be the reward parameter vector estimated by the batched UCB policy in the n -th episode. The upper bound of instantaneous regret (defined by |〈 θ n A , s i 〉 - 〈 θ ∗ A , s i 〉| ) in the FI-CBB setting is tighter than that in CBB setting (i.e., using the partial-information feedback).

From Theorem 1, we can infer that utilizing partialinformation feedbacks leads to a deterioration in the regret of the bandit policy. Ideally, the policy would be updated using full-information feedback. However, in CBB, full-information feedback is unavailable. Fortunately, in CBB, different reward parameter vectors are maintained and estimated separately for each action, and the potential reward

10

20

30

40

50

60

70

80

90

Number of Episodes N

0.236

0.238

0.240

0.242

0.244

0.246

0.248

0.250

0.252

Average Reward

partial-information feedback (CBB)

40% feedback

60% feedback

80% feedback

full-information feedback (FI-CBB)

…

…

Context

Matrix

Imputed

Context

Matrix

Reward

Vector

Imputed

Reward

Vector

B

…

B

…

…

Context Matrix

Reward

Vector

B

…

B

…

Context

Matrix

Reward

Vector

n

j

N

…

n

j

N

n

j

N

ˆ

n

j

N

n

j

N

ˆ

n

j

N

CBB with Our Reward Imputation

Full-Information CBB

CBB

j

n

A

S

j

n

A

R

j

n

A

S

ˆ

j

n

A

S

j

n

A

R

ˆ

j

n

A

R

structures of the non-executed actions have been captured to some extent. Therefore, why not utilize these maintained reward parameters to estimate the unknown rewards for the non-executed actions? In the following, we propose an efficient reward imputation approach that leverages this additional reward structure information to enhance the performance of the bandit policy.

3 Reward Imputation for Policy Updating

In this section, we present an efficient reward imputation approach tailored for policy updating in CBB setting.

Formulation of Reward Imputation. As shown in Figure 2, in contrast to CBB that ignores the contexts and rewards of the non-executed steps of each action, our reward imputation approach completes the missing values using the imputed contexts and rewards, approximating the fullinformation CBB setting. Specifically, at the end of the ( n +1) -th episode, for each action A j ∈ A , j ∈ [ M ] , the context vectors and rewards received at the steps where the action A j is executed are observed, and are stored into a context matrix S n A j ∈ R N n j × d and a reward vector R n A j ∈ R N n j , respectively, where N n j denotes the number of executed steps of A j in episode n + 1 . More importantly, at the steps (in episode n + 1 ) where the action A j is NOT executed, the following imputations need to be performed for action A j : (1) since the contexts are shared by all the actions, we directly store them into an imputed context matrix ̂ S n A j ∈ R ̂ N n j × d , where ̂ N n j denotes the number of non-executed steps of A j (i.e., ̂ N n j = B -N n j ); (2) since the rewards of A j are unobserved at the non-executed steps, we estimate them using an imputed reward vector : for any j ∈ [ M ] ,

where r n,b ( A j ) := 〈 ¯ θ n A j , s n,b 〉 denotes the imputed reward parameterized by ¯ θ n A j ∈ R d and s n,b is the b -th row of ̂ S n A j .

Next, we introduce the updating process of the reward parameter vector ¯ θ n A j . We first concatenate the context and reward matrices from the previous episodes: L n A j = [ S 0 A j ; · · · ; S n A j ] ∈ R L n j × d , T n A j = [ R 0 A j ; · · · ; R n A j ] ∈ R L n j , L n j = ∑ n k =0 N k j , ̂ L n A j = [ ̂ S 0 A j ; · · · ; ̂ S n A j ] ∈ R ̂ L n j × d , ̂ T n A j = [ ̂ R 0 A j ; · · · ; ̂ R n A j ] ∈ R ̂ L n j , ̂ L n j = ∑ n k =0 ̂ N k j . Then, the updated parameter vector ¯ θ n +1 A j of the imputed reward for action A j can be obtained by solving the following imputation regularized ridge regression : for n = 0 , 1 , . . . , N -1 ,

where γ ∈ [0 , 1] is the imputation rate that controls the degree of reward imputation and measures a trade-off between bias and variance (Remark 1&2), λ > 0 is the regularization parameter. The discounted variant of the closed least squares solution of (1) is used for computing ¯ θ n +1 A j :

and η ∈ (0 , 1) is the discount parameter that controls how fast the previous imputed rewards are forgotten, and can help guaranteeing the regret bound in Theorem 2.

Efficient Reward Imputation using Sketching. As shown in the first 4 columns in Table 1, the overall time complexity of the imputation for each action is O ( Bd 2 ) in each episode, where B represents the batch size, and d the dimensionality of the input. Thus, for all the M actions in one episode, reward imputation increases the time complexity from O ( Bd 2 ) of the approach without imputation to O ( MBd 2 ) . To address this issue, we design an efficient reward imputation approach using sketching, which reduces the time complexity of each action in one episode from O ( Bd 2 ) to O ( cd 2 ) , where c denotes the sketch size satisfying d < c < B and cd > B . Specifically, in the ( n +1) -th episode, the formulation in (1) can be approximated by a sketched ridge regression as:

where ˜ θ n +1 A denotes the updated parameter vector of the imputed reward using sketching for action A ∈ A , C n A j ∈ R c × N n j and ̂ C n A j ∈ R c × ̂ N n j are the sketch submatrices for the observed term and the imputation term, respectively, and the sketch matrices for the two terms can be represented as

We denote the sketches of the context matrix and the reward vector by Γ n A j := C n A j S n A j ∈ R c × d and Λ n A j := C n A j R n A j ∈ R c , the sketches of the imputed context matrix and the imputed reward vector by ̂ Γ n A j := ̂ C n A j ̂ S n A j ∈ R c × d and ̂ Λ n A j := ̂ C n A j ̂ R n A j ∈ R c . Similarly to (2), the discounted variant of the closed solution of (5) as follows:

where η ∈ (0 , 1) denotes the discount parameter, W n +1 A j := λ I d + G n +1 A j + γ ̂ G n +1 A j , and

Using the parameter ˜ θ n +1 A j , we obtain the sketched version of imputed reward as ˜ r n,b ( A j ) := 〈 ˜ θ n A j , s n,b 〉 at step b ∈ [ ̂ N n j ] . Finally, we specify that the sketch submatrices { C n A } A ∈A ,n ∈ [ N ] and { ̂ C n A j } A ∈A ,n ∈ [ N ] are the block construction of Sparser Johnson-Lindenstrauss Transform (SJLT) (Kane and Nelson, 2014), where the sketch size c is divisible by the number of blocks D 3 . As shown in the last 4 columns in Table 1, sketching reduces the time complexity of reward imputation from O ( MBd 2 ) to O ( Mcd 2 ) for all M actions in one episode, where c < B . When Mc ≈ B , the overall time complexity of our reward imputation using sketching is even comparable to that without reward imputation, i.e., a O ( Bd 2 ) time complexity.

Updated Policy using Imputed Rewards. Inspired by the UCB strategy (Li et al., 2010), the updated policy for online decision of the ( n +1) -th episode can be formulated using the imputed rewards (parameterized by ¯ θ n +1 A in (2)) or the sketched version of imputed rewards (parameterized by ˜ θ n +1 A in (6)). Specifically, for a new context s ,

sketched policy ˜ p n +1 selects the action as A ← arg max A ∈A 〈 ˜ θ n +1 A , s 〉 + α [ s ᵀ ( W n +1 A ) -1 s ] 1 2 , where ω ≥ 0 and α ≥ 0 are the regularization parameters in policy and their theoretical values are given in Theorem 4. We summarize the reward imputation using sketching and the sketched policy into Algorithm 2, called SPUIR. Similarly, we call the updating of the original policy that uses reward imputation without sketching, the Policy Updating with Imputed Rewards (PUIR).

origin policy ¯ p n +1 selects the action as A ← arg max A ∈A 〈 ¯ θ n +1 A , s 〉 + ω [ s ᵀ ( Ψ n +1 A ) -1 s ] 1 2 ,

Algorithm 2 Sketched Policy Updating with Imputed Rewards (SPUIR) in the ( n +1) -th episode

INPUT: Policy ˜ p n , data buffer D n +1 , A = { A j } j ∈ [ M ] , α ≥ 0 , η ∈ (0 , 1) , γ ∈ [0 , 1] , λ > 0 , W 0 A j = λ I d , G 0 A j = ̂ G 0 A j = O d , p 0 A j = ˆ p 0 A j = 0 , ˜ θ 0 A j = 0 , j ∈ [ M ] , batch size B , sketch size c , number of block D OUTPUT: Updated policy ˜ p n +1

1: For all j ∈ [ M ] , store context vectors and rewards corresponding to the steps at which the action A j is executed, into Γ n A j ∈ R N n j × d and Λ n A j ∈ R N n j

2: For all j ∈ [ M ] , store context vectors corresponding to the steps at which the action A j is not executed into ̂ Γ n A j ∈ R ̂ N n j × d , where ̂ N n j ← B -N n j

3: ˜ r n,b ( A j ) ←〈 ˜ θ n A j , s n,b 〉 , for all A j ∈ A and b ∈ [ ̂ N n j ] , where s n,b is the b -th row of ̂ Γ n A j

4: Compute imputed reward vector ̂ R n A j ←{ ˜ r n, 1 ( A j ) , . . . , ˜ r n, ̂ N n j ( A j ) } ∈ R ̂ N n j , ∀ j ∈ [ M ]

5: for all action A j ∈ A do

6: G n +1 A j ← G n A j + Γ n ᵀ A j Γ n A j , p n +1 A j ← p n A j + Γ n ᵀ A j Λ n A j {(7)} ̂ G n +1 A j ← η ̂ G n A j + ̂ Γ n ᵀ A j ̂ Γ n A j , ˆ p n +1 A j ← η ˆ p n A j + ̂ Γ n ᵀ A j ̂ Λ n A j {(8)}

7: W n +1 A j ← λ I d + G n +1 A j + γ ̂ G n +1 A j , ˜ θ n +1 A j ← ( W n +1 A j ) -1 ( p n +1 A j + γ ˆ p n +1 A j ) {(6)}

8: end for

9: ˜ p n +1 ( s ) selects action A ← arg max A ∈A 〈 ˜ θ n +1 A , s 〉 + α [ s ᵀ ( W n +1 A ) -1 s ] 1 2 for a new context s 10: Return { ˜ θ n +1 A } A ∈A , { ( W n +1 A ) -1 } A ∈A

4 Theoretical Analysis

We provide the instantaneous regret bound, prove the approximation error of sketching, and analyze the regret of SPUIR in CBB setting. The detailed proofs can be found in Appendix B. We first demonstrate the instantaneous regret bound of the original solution ¯ θ n A in (1).

Theorem 2 (Instantaneous Regret Bound) . Let η ∈ (0 , 1) be the discount parameter, γ ∈ [0 , 1] the imputation rate. In the n -th episode, if the rewards { R n,b } b ∈ [ B ] are independent 4 and bounded by C R , then, for any b ∈ [ B ] , ∀ A ∈ A , there exists C Imp > 0 such that, with probability at least 1 -δ ,

where Ψ n A = λ I d + Φ n A + γ ̂ Φ n A , ν = [2 C 2 R log(2 MB/δ )] 1 2 . The first term on the right-hand side of (9) can be seen as the bias term for the reward imputation, while the second term is the variance term. The variance term of our algorithm is not larger than that without the reward imputation, i.e, for any s ∈ R d ,

Further, a larger imputation rate γ leads to a smaller variance term [ s ᵀ ( Ψ n A ) -1 s ] 1 2 .

Remark 1 (Smaller Variance) . From Theorem 2, we can observe that our reward imputation achieves a smaller variance ( [ s ᵀ n,b ( Ψ n A ) -1 s n,b ] 1 2 ) than that without the reward imputation. By combining Theorem 2 and the proof of Theorem 1, we can obtain that the variance in instantaneous regret bound of SPUIR is in between the variances in full and partial information scenarios. Thus, reward imputation in SPUIR provides a promising way to use expert advice approaches for bandit problems.

Remark 2 (Controllable Bias) . Our reward imputation approach incurs a bias term γ 1 2 η -1 2 C Imp in addition to the two bias terms λ ‖ θ ∗ A ‖ 2 and ν that exist in every existing UCB-based policy. But the additional bias term γ 1 2 η -1 2 C Imp is controllable due to the presence of imputation rate γ that can help controlling the additional bias. Moreover, the term C Imp in the additional bias can be replaced by a function f Imp ( n ) , and f Imp ( n ) is monotonic decreasing w.r.t. number of episodes n provided that the mild condition √ η = Θ( d -1 ) holds (the definition and analysis about f Imp can be found in Appendix B.1). Overall, the imputation rate γ controls a trade-off between the bias term and the variance term, and we will design a rate-scheduled approach for automatically setting γ in Section 5.

Remark 3 (Relationship with Existing Instantaneous Regrets) . According to the original definition in the context of online learning, the definition of instantaneous regret should be max A ∈A 〈 θ ∗ A , s n,b 〉 -〈 θ ∗ A I n,b , s n,b 〉 . However, in the specific setting of contextual batched bandit (CBB) that is the focus of this paper, as derived in Appendix (second inequality of Eq. (48)), if we denote the upper bound of |〈 ¯ θ n A , s n,b 〉-〈 θ ∗ A , s n,b 〉| as U, then 2U serves as an upper bound for instantaneous regret. Thus, in the context of CBB explored in this paper, we are interested in an upper bound for |〈 ¯ θ n A , s n,b 〉-〈 θ ∗ A , s n,b 〉| and define it as the instantaneous regret bound.

Although some approximation error bounds using SJLT have been proposed (Nelson and Nguyên, 2013; Kane and Nelson, 2014; Zhang and Liao, 2019), it is still unknown what is the lower bound of the sketch size while applying SJLT to the sketched ridge regression problem in our SPUIR. Next, we prove the approximation error as well as the lower bound of the sketch size in SPUIR. For convenience, we drop all the superscripts and subscripts in this result.

Theorem 3 (Approximation Error Bound of Imputation using Sketching) . Denote the imputation regularized ridge regression function by F ( θ ) (defined in (1) ) and the sketched ridge regression function by F S ( θ ) (defined in (5) ) for reward imputation, whose solutions are ¯ θ = arg min θ ∈ R d F ( θ ) and ˜ θ = arg min θ ∈ R d F S ( θ ) . Let γ ∈ [0 , 1] be the imputation rate, λ > 0 the regularization parameter, δ ∈ (0 , 0 . 1] , ε ∈ (0 , 1) , L all = [ L ; √ γ ̂ L ] , and ρ λ = ‖ L all ‖ 2 2 / ( ‖ L all ‖ 2 2 + λ ) . If Π and ̂ Π are SJLT, assuming that D = Θ( ε -1 log 3 ( dδ -1 )) and the sketch size c = Ω ( d polylog ( dδ -1 ) /ε 2 ) , with probability at least 1 -δ, the following results hold:

To measure the convergence of approximating the optimal policy in an online manner, we define the regret of SPUIR against the optimal policy as

where I n,b denotes the index of the executed action using the sketched policy ˜ p n (parameterized by { ˜ θ n A } A ∈A ) at step b in the n -th episode. We final prove the regret bound of SPUIR.

Theorem 4 (Regret Bound of SPUIR) . Let T = BN be the overall number of steps, η ∈ (0 , 1) be the discount parameter, γ ∈ [0 , 1] the imputation rate, λ > 0 the regularization parameter, C max θ ∗ = max A ∈A ‖ θ ∗ A ‖ 2 , C Imp be the positive constant defined in Theorem 2. Assume that the conditional independence assumption in Theorem 2 holds and the upper bound of rewards is C R , M = O (poly( d )) , T ≥ d 2 , ν = [2 C 2 R log(2 MB/δ 1 )] 1 2 with δ 1 ∈ (0 , 1) , ω = λC max θ ∗ + ν + γ 1 2 η -1 2 C Imp , α = ωC α , where C α > 0 which decreases with increase of 1 /ε and ε ∈ (0 , 1) . Let δ 2 ∈ (0 , 0 . 1] , ρ λ < 1 be the constant defined in Theorem 3, and C reg be a positive constant that decreases with increase of 1 /ε . For the sketch matrices { Π n A } A ∈A ,n ∈ [ N ] and { ̂ Π } A ∈A ,n ∈ [ N ] , assuming that the number of blocks in SJLT D = Θ( ε -1 log 3 ( dδ -1 2 )) , and the sketch size satisfying

then, for an arbitrary sequence of contexts { s n,b } n ∈ [ N ] ,b ∈ [ B ] , with probability at least 1 -N ( δ 1 + δ 2 ) ,

Remark 4. Setting B = O ( √ T/d ) , ρ λ glyph[epsilon1] = 1 /d yields a sublinear regret bound of order ˜ O ( MdT ) 5 provided that the sketch size c = Ω( ρ 2 λ d 3 polylog( dδ -1 2 )) . We can observe that the lower bound of c is independent of the overall number of steps T , and a theoretical value of the batch size is B = C B √ T/d = C 2 B N/d , where setting C B ≈ 25 is a suitable choice that has been verified in the experiments in Section 6. In particular, when ρ λ = O (1 /d ) , the sketch size of order c = Ω( d polylog d ) is sufficient to achieve a sublinear regret.

From the theoretical results of regret, we can observe that our SPUIR admits several advantages: (a) The order of our regret bound (w.r.t. the overall number of steps) is not higher than those in the literature in the fully-online setting (Li et al., 2019; Dimakopoulou et al., 2019) that is a more simple setting than ours; (b) The first term in the regret bound (10) measures the performance of policy updating using imputed rewards (called 'policy error'). From Theorem 2 and Remark 1&2, we obtain that, in each episode, our policy updating has a smaller variance than the policy without the reward imputation, and incurs a decreasing additional bias under mild conditions, leading to a tighter regret (i.e., smaller policy error) after some number of episodes. (c) The second term on the right-hand side of (10) is of order O ( T √ ρ λ glyph[epsilon1]d/B ) , which is incurred by the sketching approximation using SJLT (called 'sketching error'). This sketching error does not have any influence on the order of regret of SPUIR, which may even have a lower order with a suitable choice of ρ λ ε , e.g., setting ρ λ ε = T -1 / 4 d -1 yields a sketching error of order O ( T 3 / 8 d 1 / 2 ) provided that c = Ω( ρ 2 λ d 3 polylog( dδ -1 2 ) √ T ) .

At a fundamental level, the effectiveness of the proposed reward imputation can be attributed to the following two key factors:

(1) Leveraging contextual bandit structure : Traditional bandit methods only consider the structural assumptions for executed actions, leaving out non-executed ones. Our reward imputation approach incorporates a wide range of reward function structural assumptions, covering both executed and non-executed actions. By imputing missing rewards with observed data, we reduce the impact of missing data for a more accurate reward estimation.

(2) Balancing exploration and exploitation : Reward imputation's effectiveness arises from its impact on the exploration-exploitation trade-off. By incorporating imputed rewards, our proposed algorithms can make informed decisions even when observed rewards are incomplete. This enhances the agent's exploration strategy, helping it discover more valuable actions and reducing cumulative regret. Essentially, our reward computation approach approximates full-information feedback, mitigating the explore/exploit dilemma.

5 Extensions of Our Approach

To make the proposed reward imputation approach more feasible and practical, we tackle the following two research questions by designing variants of our approach following the theoretical results:

RQ1 (Parameter Selection): Can we set the imputation rate γ without tuning?

RQ2 (Nonlinear Reward): Can we apply the proposed reward imputation approach to the case where the expectation of true rewards is nonlinear?

Rate-Scheduled Approach. For RQ1, we equip PUIR and SPUIR with a rate-scheduled approach, called PUIR-RS and SPUIR-RS, respectively. From Remark 1&2, a larger imputation rate γ leads to a smaller variance while increasing the bias, while the bias term includes a monotonic decreasing function w.r.t. number of episodes under mild conditions. Therefore, we can gradually increase γ with the number of episodes, avoiding the large bias at the beginning of reward imputation. Specifically, we set γ = X % for episodes from ( X -10)% × N to X % × N , where X ∈ [10 , 100] .

Application to Nonlinear Rewards. For RQ2, we provide nonlinear versions of reward imputation. We use linearization technologies of nonlinear rewards, rather than directly setting the rewards as nonlinear functions (Valko et al., 2013; Chatterji et al., 2019), avoiding the linear regret or curse of

kernelization. Specifically, instead of using the linear imputed reward ˜ r n,b ( A j ) := 〈 ˜ θ n A j , s n,b 〉 , we use the following linearized nonlinear imputed rewards, denotes by T n,b ( θ , A ) :

(1) SPUIR-Exp. We assume that the expected reward is an exponential function as G E ( θ , s ) = exp( θ ᵀ s ) . Then T n,b ( θ , A ) = 〈 θ , ∇ θ G E ( θ , s n,b ) 〉 , where ∇ θ G E ( θ , s n,b ) = exp ( θ ᵀ s n,b ) s n,b .

(2) SPUIR-Poly. When the expected reward is a polynomial function as G P ( θ , s ) = ( θ ᵀ s ) 2 . Then T n,b ( θ , A ) = 〈 θ , ∇ θ G P ( θ , s n,b ) 〉 , where ∇ θ G P ( θ , s n,b ) = 2 ( θ ᵀ s n,b ) s n,b .

(3) SPUIR-Kernel. Consider that the underlying expected reward in a Gaussian reproducing kernel Hilbert space (RKHS). We use T n,b ( θ , A ) = 〈 θ , φ ( s n,b ) 〉 in random feature space, where the random feature mapping φ can be explicitly computed.

For SPUIR-Exp and SPUIR-Poly, combining the linearization of convex functions (Shalev-Shwartz, 2011) with Theorem 4 yields the regret bounds of the same order. For SPUIR-Kernel, using the approximation error of random features (Rahimi and Recht, 2008), we can also obtain that, SPUIRKernel has the same regret bound as SPUIR under mild conditions (see proofs in Appendix B).

6 Experiments

We empirically evaluated the performance of our algorithms on 3 datasets: the synthetic dataset, publicly available Criteo dataset 6 ( Criteo-recent , Criteo-all ), and dataset collected from Tencent's WeChat app for coupon recommendation ( commercial product ).

Experimental Settings. We compared our algorithms with: Sequential Batch UCB (SBUCB) (Han et al., 2020), Batched linear EXP3 (BEXP3) (Neu and Olkhovskaya, 2020), Batched linear EXP3 using Inverse Propensity Weighting (BEXP3-IPW) (Bistritz et al., 2019), Batched Balanced Linear Thompson Sampling (BLTS-B) (Dimakopoulou et al., 2019), and Sequential version of Delayed Feedback Model (DFM-S) (Chapelle, 2014). We applied the algorithms to CBB setting and implemented on Intel(R) Xeon(R) Silver 4114 CPU@2.20GHz, and repeated the experiments 20 times. We tested the performance of algorithms in streaming recommendation scenarios, where the reward is represented by a linear combination of the click and conversion behaviors of users. According to Remark 4, we set the batch size as B = C 2 B N/d , the constant C B ≈ 25 , and the sketch size c = 150 on all the datasets. The average reward was used to evaluate the accuracy of algorithms.

Performance Evaluation. Figure 3(a)-(c) reports the average reward of SPUIR with its variants and the baselines. We observed that SPUIR and its variants achieved higher average rewards, demonstrating the effectiveness of our reward imputation. Moreover, SPUIR and its rate-scheduled version SPUIR-RS had similar performances compared with the origin PUIR, which indicates the practical effectiveness of our variants and verifies the correctness of the theoretical analyses. The results on commercial product in Table 2 indicate that SPUIR outperformed the second-best baseline with the improvements of 1.07% CVR (conversion rate) and 1.12% CTCVR (post-view

(⋃(∑(⊎(∮(∫(∖(∫(⋃

(((/((⌋(

















⊗√⌉̂˜√(}˜(〉√]√}̂˜√(⊗

/

/

/

/

/

/

/

√˜√∐˜˜(∫˜glyph[arrowvertexdbl]∐√̂

∑⊎∮∫

⋃∑⊎∮∫

∑⊎∮∫∖∫⋃

⋃∑⊎∮∫∖∫⋃

⊕⋂⋃∖

⋃⊎

(a)

synthetic

data













⊗√⌉̂˜√(}˜(〉√]√}̂˜√(⊗

/

/

/

/

/

√˜√∐˜˜(∫˜glyph[arrowvertexdbl]∐√̂

∑⊎∮∫

⋃∑⊎∮∫

∑⊎∮∫∖∫⋃

⋃∑⊎∮∫∖∫⋃

⊕⋂⋃∖

⋃⊎

〉∑∑∖∮∑∨

〉∑∑

(b)

Criteo-all













⊗√⌉̂˜√(}˜(〉√]√}̂˜√(⊗

/

/

/

/

/

√˜√∐˜˜(∫˜glyph[arrowvertexdbl]∐√̂

∑⊎∮∫

⋃∑⊎∮∫

∑⊎∮∫∖∫⋃

⋃∑⊎∮∫∖∫⋃

⊕⋂⋃∖

⋃⊎

(c)

commercial

product











⊗√⌉̂˜√(}˜(〉√]√}̂˜√(⊗

/

/

/

/

/

/

/

√˜√∐˜˜(∫˜glyph[arrowvertexdbl]∐√̂

⋃∑⊎∮∫

⋃∑⊎∮∫∖〉glyph[arrowtp]√

⋃∑⊎∮∫∖∑}⌈glyph[arrowbt]

⋃∑⊎∮∫∖⊙˜√{˜⌈

(d) nonlinear reward

(((⌋(⊗

((⌋(⊗

((⌋(⊗

((⌋(⊗

((⌋(⊗

((⌋(⊗

((⌋(⊗

((⌋(⊗

((⌋(⊗ (⊗

(⊗

((/(((

((/(((

((/(((

((/(((

((/(((

((/(((

((√(˜(√(∐(˜(˜(((∫(˜(glyph[arrowvertexdbl](∐(√(̂

(((((

(((((

((((((

((((((

((((((

(e) batch size

B

((/((⌋(

((/((⌋(

(((/((⌋(

(((/((⌋(

(((/((⌋(

(((/((⌋(

(((/((⌋(

(((/((⌋(

(((/((⌋(

(((/((⌋(

(⋃(⌋(˜(√(̂([(((⋃(](︷(˜(((̂

((/(((

((/(((

((/(((

((/(((

((/(((

((√(˜(√(∐(˜(˜(((∫(˜(glyph[arrowvertexdbl](∐(√(̂

(⋃(∑(⊎(∮(∫

(f) sketch size

c

click-through&conversion rate). Besides, our reward imputation approaches were more efficient than DFM-S, BLTS-B. The variants using sketching of our algorithms (SPUIR, SPUIR-RS) significantly reduced the time costs of reward imputation, and took less than twice as long to run compared to the baselines without reward imputation (SBUCB, BEXP3, BEXP3-IPW). Figure 3(d) illustrates performances of SPUIR and its nonlinear variants, where SPUIR-Kernel achieved the highest rewards indicating the effectiveness of the nonlinear generalization of our approach. For different decision tasks, a suitable nonlinear reward model needs to be selected for better performances.

Parameter Influence. From the regret bound (10), we can observe that a larger batch size B results in a larger first term (of order O ( B ) , called policy error) but a smaller second term (of order O (1 /B ) , called sketching error), indicating that a suitable batch size B needs to be set. This conclusion was empirically verified in Figure 3(e), where B = 1 , 000 ( C B = 25 ) yields better empirical performance in terms of the average reward. Similar phenomenon can also be observed on Criteo dataset and commercial product . All of the results verified the theoretical results in Remark 4: B = C B √ T/d = C 2 B N/d is a suitable choice while setting C B ≈ 25 . From the results in Figure 3(f) we observe that, for our SPUIR and SPUIR-RS, the performances significantly increased when the sketch size c reached 10% B ( ≈ d log d ), which demonstrates the conclusion in Remark 4 that only the sketch size of order c = Ω( d polylog d ) is needed for satisfactory performance.

7 Conclusion

This paper presents a computationally efficient reward imputation approach for contextual batched bandits that addresses the challenge of partial-information feedback in real-world applications. The proposed approach mimics the reward generation mechanism of the environment, approximating full-information feedback. It reduces time complexity using sketching, achieves a relative-error bound for approximation, and exhibits regret with controllable bias and reduced variance. The theoretical formulation and algorithmic implementation may provide an efficient reward imputation scheme for online learning under limited feedback.

Acknowledgments and Disclosure of Funding

This work was funded by the National Key R&D Program of China (2022ZD0114802), the National Natural Science Foundation of China (62376275, U2001212, 62377044), Intelligent Social Governance Interdisciplinary Platform, Major Innovation & Planning Interdisciplinary Platform for the 'Double-First Class' Initiative, Renmin University of China. Supported by fund for building world-class universities (disciplines) of Renmin University of China. Supported by Public Computing Cloud, Renmin University of China. Supported by the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (23XNKJ13).

References

Auer, P. (2002). Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3:397-422.

Bistritz, I., Zhou, Z., Chen, X., Bambos, N., and Blanchet, J. (2019). Online EXP3 learning in adversarial bandits with delayed feedback. In Advances in Neural Information Processing Systems 32 , pages 11349-11358.

Calandriello, D., Lazaric, A., and Valko, M. (2017). Efficient second-order online kernel learning with adaptive embedding. In Advances in Neural Information Processing Systems 30 , pages 6140-6150.

Chapelle, O. (2014). Modeling delayed feedback in display advertising. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1097-1105.

Chatterji, N. S., Pacchiano, A., and Bartlett, P. L. (2019). Online learning with kernel losses. In Proceedings of the 36th International Conference on Machine Learning , pages 971-980.

Chou, K.-C., Lin, H.-T., Chiang, C.-K., and Lu, C.-J. (2015). Pseudo-reward algorithms for contextual bandits with linear payoff functions. In Proceedings of the 6th Asian Conference on Machine Learning , pages 344-359.

Chu, W., Li, L., Reyzin, L., and Schapire, R. E. (2011). Contextual bandits with linear payoff functions. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics , pages 208-214.

Dimakopoulou, M., Zhou, Z., Athey, S., and Imbens, G. (2019). Balanced linear contextual bandits. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence , pages 3445-3453.

Esfandiari, H., Karbasi, A., Mehrabian, A., and Mirrokni, V. S. (2021). Regret bounds for batched bandits. In Proceedings of the 35th AAAI Conference on Artificial Intelligence , pages 7340-7348.

Gao, Z., Han, Y., Ren, Z., and Zhou, Z. (2019). Batched multi-armed bandits problem. In Advances in Neural Information Processing Systems 32 , pages 501-511.

Gu, Q., Karbasi, A., Khosravi, K., Mirrokni, V., and Zhou, D. (2021). Batched neural bandits. arXiv preprint arXiv:2102.13028 .

Han, Y., Zhou, Z., Zhou, Z., Blanchet, J. H., Glynn, P. W., and Ye, Y. (2020). Sequential batch learning in finite-action linear contextual bandits. CoRR , abs/2004.06321.

Jin, C., Allen-Zhu, Z., Bubeck, S., and Jordan, M. I. (2018). Is q-learning provably efficient? In Advances in Neural Information Processing Systems 31 .

Kane, D. M. and Nelson, J. (2014). Sparser Johnson-Lindenstrauss transforms. Journal of the ACM , 61(1):4:14:23.

Kim, G. and Paik, M. C. (2019). Doubly-robust lasso bandit. In Advances in Neural Information Processing Systems 32 , pages 5869-5879.

Lan, A. S. and Baraniuk, R. G. (2016). A contextual bandits framework for personalized learning action selection. In Proceedings of the 9th International Conference on Educational Data Mining , pages 424-429.

Li, L., Chu, W., Langford, J., and Schapire, R. E. (2010). A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web , pages 661-670.

Li, S., Chen, W., Li, S., and Leung, K. (2019). Improved algorithm on online clustering of bandits. In Kraus, S., editor, Proceedings of the 28th International Joint Conference on Artificial Intelligence , volume Li2019Improved, pages 2923-2929.

Nelson, J. and Nguyên, H. L. (2013). OSNAP: Faster numerical linear algebra algorithms via sparser subspace embeddings. In Proceedings of the 54th Annual Symposium on Foundations of Computer Science , pages 117-126.

Neu, G. and Olkhovskaya, J. (2020). Efficient and robust algorithms for adversarial linear contextual bandits. In Proceedings of the 33rd Conference on Learning Theory , pages 3049-3068.

Perchet, V., Rigollet, P., Chassang, S., and Snowberg, E. (2016). Batched bandit problems. The Annals of Statistics , 44(2):660-681.

Rahimi, A. and Recht, B. (2008). Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In Advances in Neural Information Processing Systems 21 , pages 1313-1320.

Ren, Z. and Zhou, Z. (2020). Dynamic batch learning in high-dimensional sparse linear contextual bandits. arXiv preprint arXiv:2008.11918 .

Shalev-Shwartz, S. (2011). Online learning and online convex optimization. Foundations and Trends R © in Machine Learning , 4(2):107-194.

Valko, M., Korda, N., Munos, R., Flaounas, I. N., and Cristianini, N. (2013). Finite-time analysis of kernelised contextual bandits. In Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence .

Wang, C. and Cheng, G. (2020). Online batch decision-making with high-dimensional covariates. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics , pages 3848-3857.

Woodruff, D. P. (2014). Sketching as a tool for numerical linear algebra. Foundations and Trends R © in Theoretical Computer Science , 10(1-2):1-157.

Yang, J., Hu, W., Lee, J. D., and Du, S. S. (2021). Impact of representation learning in linear bandits. In Proceedings of the 9th International Conference on Learning Representations .

Yom-Tov, E., Feraru, G., Kozdoba, M., Mannor, S., and Hochberg, I. (2017). Encouraging physical activity in patients with diabetes: Intervention using a reinforcement learning system. Journal of Medical Internet Research , 19(10):e338.

Zhang, K. W., Janson, L., and Murphy, S. A. (2020). Inference for batched bandits. In Advances in Neural Information Processing Systems 33 , pages 9818-9829.

Zhang, X., Dai, S., Xu, J., Dong, Z., Dai, Q., and Wen, J. (2022). Counteracting user attention bias in music streaming recommendation via reward modification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2504-2514.

Zhang, X., Jia, H., Su, H., Wang, W., Xu, J., and Wen, J. (2021). Counterfactual reward modification for streaming recommendation with delayed feedback. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 41-50.

Zhang, X. and Liao, S. (2019). Incremental randomized sketching for online kernel learning. In Proceedings of the 36th International Conference on Machine Learning , pages 7394-7403.

Zhou, Z. (2023). Stream efficient learning. CoRR , abs/2305.02217.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on contextual batched bandit (CBB) problems, where only partial information (the rewards of executed actions) is available. The authors proposed an efficient reward imputation approach in CBB, which completes the unobserved reward, and the new objective can be solved by the imputation regularized ridge regression. They further proposed sketching to reduce time complexity. Theoretical guarantees are established, showing that this approach has a sublinear regret bound against the optimal policy.  Experiments are conducted, showing its superiority over baselines.\n\nSTRENGTHS:\nThe paper is well-structured. The proposed reward imputation for the non-executed actions is innovative, and this idea seems widely applicable in many scenarios. Proof-of-concept experimental results support their theoretical claims.\n\nWEAKNESSES:\nA high-level idea of reward imputation needs to be provided, especially regarding why reward imputation is effective statistically. At a high level, it seems counter-intuitive since the imputed rewards are derived from the observed rewards, so it should not yield any statistical improvement. If there is indeed no improvement, the position of this work would be in question. However, the authors did achieve improvement, so it would be valuable for them to offer a high-level explanation regarding why reward imputation works in this context.\n\nIn addition, the absence of definitions for certain notations in the paper somewhat makes it challenging to evaluate certain results. I apologize if I may have overlooked these definitions in the paper. I have listed these issues in the \"Questions\" section below.\n\nQUESTIONS:\n- Is the batch size $B$ predetermined by the problem setup, or can it be manually adjusted by the learner? In the analysis conducted by the authors (e.g., Remark 3), they define $B$ to be on the order of $O(\\sqrt{T})$, indicating that it is likely the latter case. In that context, what are the theoretical advantages of this algorithm when setting $B=1$ compared to standard contextual bandit algorithms? Can we simply apply CB algorithms to this particular setting with $B=1$ then?\n\n- It looks like eq. (2) is a solution to eq. (1), which makes me wonder where the discount parameter $\\eta$ in line 156 comes from. Is $\\eta$ a predetermined hyperparameter or an arbitrary parameter? If it is the former, I did not find it in the problem setup; if it is the latter, it seems so weird as it suggests that any $\\eta$ would make eq. (2) a solution to eq. (1)? \n\n- Why is Theorem 2 referred to as the \"instantaneous regret bound\"? Shouldn't the instantaneous regret involve $\\max\\_a\\langle\\theta^*\\_A,s\\_{n,b}\\rangle$? This theorem seems to be more of a bound on estimation error.\n\nLIMITATIONS:\nNo potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses imputation to improve the learning of the reward parameters, even for nonexecuted steps. This results in better variance and empirical benefits over many existing algorithms.\n\nSTRENGTHS:\n1. The empirical benefit seems quite strong over existing algorithms on real datasets.\n2. The proposed idea is clear and intuitive.\n\nWEAKNESSES:\n1. I found the notation quite difficult to parse, especially at the beginning of page 4. For example, $L_j^n$ was discussed before it was defined.  Moreover, what is $\\nu$ in Theorem 2?\n2. The improvement seen by imputation is a little tricky to see without knowledge of $C_{\\text{imp}}$ in the bound. I find it a little dubious that the amount in which the bias increases by using imputation is hidden by this mysterious constant $C_{\\text{imp}}$. The fact that it is monotonically decreasing is not sufficient. I would like to see a more in-depth analysis of this term to understand better whether imputation truly improves our bounds. \n3. A similar problem is in Theorem 4. The constant $C_{\\text{reg}}$ is presented without a true interpretation. The improvement of the regret seen here over traditional algorithms is unclear. I believe the dependence on $d$ and $T$ achieved here are similar to what is achieved in the literature already. Therefore, understanding the other constants here is important.\n\nQUESTIONS:\n1. I think the paper could be improved by reducing the amount of time discussing the difference between full and partial feedback. I don't believe it adds much to the paper. \n2. How does the regret analysis in Theorem 4 compare to recent algorithms tackling Contextual Batched Bandits?\n\nLIMITATIONS:\nThe limitations could have been a bit more developed and further extensions could be discussed more. \n\nOverall, I currently vote for acceptance. However, if the authors do not provide a good rebuttal and fix the issues with the limitations, I may have to reduce my score. However, I believe the issues are fixable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the contextual batched bandit (CBB) problem, where the learner interacts with the environment in an online manner through $N$ episodes. In episode $n$, the learner selects the policy $p_n$,  receives $B$ contexts $s_{n,b}\\in \\mathbb{R}^d$,  decides the action $A_{I_{n,b}}$ according to $p_n(s_{n,b})$, receives the feedback and updates the policy. The reward is determined by an unknown action-specific reward parameter vector $\\theta_{A}^\\star$ such that $\\mathbb{E}[ R_{i,A} ] = \\langle \\theta_{A}^\\star, s_i  \\rangle$ for context $s_i$. \n\nTheorem 1 states the fact that the upper bound of instantaneous regret in the FI-CBB setting is tighter than that in CBB setting, where \"FI\" means full-information feedback, that is, the learner is able to observe all the losses of both executed and non-executed actions. \n\nThen, this paper proposes reward imputation to estimate the reward feedbacks for the non-executed actions. Roughly speaking, the learner estimates the reward for the non-executed actions by using an imputed reward vector in Line 144. Then, the parameter vector is updated of the imputed reward and the observed reward together (with respect to $\\gamma$, the imputation rate) as shown in Equation. (1). To further improve the efficiency, the paper utilizes the sketching technique to reduce the computational cost from $\\mathcal{O}(Bd^2)$ to $\\mathcal{O}(cd^2)$ where $c$ is the sketch size and assumed to be $d<c<B$ and $dc>B$. \n\nThe instantaneous regret bound is stated in Theorem 2. The approximation error bound of using sketching is described in the Theorem 3, and the final regret bound of the reward imputation with sketching is stated in Theorem 4.\n\nSTRENGTHS:\n1. The paper is clearly-written and self-contained. The notations are well-defined prior to the usages. \n2. The paper has some nice experiment results for demonstration.\n\nWEAKNESSES:\n1. The comparison between the regret bound with/without reward imputation is a little vague: from line 248 to line 250, the authors claim that \"in each episode, our policy updating has a smaller variance than the policy without the reward imputation, and incurs a decreasing additional bias under mild conditions, leading to a tighter regret (i.e., smaller policy error) after some number of episodes\". I wonder if the authors could be more specific that which algorithm they are comparing to? \n2. The assumption \"if $\\Pi$ and $\\widehat{\\Pi}$ are SJLT, assuming that $D=\\Theta(...)$\" is not very clear.  Does this assumption hold usually or with high-probability? \n3. Is the batch size $B$ a tunable hyper-parameter? In Remark 3, the authors set $B = \\mathcal{O}(\\sqrt{T/d})$, while in Protocol 1, the batch size $B$ seems to be fixed prior to the interaction? In addition, the condition that $d<c<B$ and $cd>B$ is not reasoned in details. \n\n[Minor Issues] \n1. The problem setting is not well-organized. The interaction protocol is described separately in the introduction and problem formulation sections. I would like to suggest putting all the notations into one section for better readability.\n\nQUESTIONS:\nMy questions are raised in the weakness section. I am willing to re-evaluate the score if the questions are answered properly.\n\nLIMITATIONS:\nThis work is pure theoretical, and does not have any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the contextual batched bandits (CBB) setting, where compared to classical online CB, a batch of actions is played and the reward being observed in an episode. Motivated from the benefit of full-information feedback, it tries to perform the reward imputation for unlabelled data, and optimizes the reward parameter by minimizing the combined MSE loss of observed data and imputed data. To further reduce computational complexity, it uses sketching to compress the matrix used in linear regression. Empirically, it shows better performance in both synthetic and real-world experiments.\n\nSTRENGTHS:\n- In the batch bandit literature, the bias-variance trade-off is mostly happened between IPS and reward modeling approaches. This paper focus on another level of bias-variance trade-off within the reward model estimate. It performs reward imputation on unlabelled data then doing regression on top of that. The overall objective contains both the observed fitting loss and the weighted imputed loss, where increasing the weight increases the bias, but potentially decreasing the variance, which gives another level of bias-variance trade-off within the reward model. The idea is interesting.\n\n- The paper is well-written and easy to follow. The method is supported by the theoretical analysis of the instantaneous regret bound. \n\n- To further reduce the complexity, the paper introduces using sketching based on the SJLT, which significantly reduces the time complexity, without losing too much performance.\n\nWEAKNESSES:\n- Intuitively, the imputed data in early rounds seem less important (i.e., in terms of accuracy) compared with the imputed data in later rounds, which this requires we basically weigh the imputed data differently, instead of using a fixed $gamma$ for all imputed data. I am not sure how the authors think about this aspect? It would be interesting to see how it works empirically.\n\n- The imputed method seems a way to improve reward models. Fitting a reward model is like a supervised learning problem and there are bunch of ways to reduce variance, like ensemble-based, it would be great to add these baselines, and arguing how the current method helps.\n\n- It seems from Eq 9, the $\\gamma$ term could be optimized to minimize the instantaneous regret bound, instead of using a heuristics on this.\n\n- Given there exists synthetic experiments, is it possible to plot the bias, variance, regret for each method? and visualize how the PUIR/SPUIR helps, and how they support the theory.\n\n- I am actually curious to see whether this way of improving reward model also translates to doubly robust based method, where it combines with IPS weighting based approach?\n\nQUESTIONS:\nSee weakness part.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}