SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- origin policy  $\bar{p}_{n+1}$  selects the action as  $A \gets \arg \max_{A \in \mathcal{A}} \langle \bar{\theta}_A^{n+1}, s \rangle + \omega[s^{\mathsf{T}}(\Psi_A^{n+1})^{-1}s]^{\frac{1}{2}}$ ,
- sketched policy  $\tilde{p}_{n+1}$  selects the action as  $A \gets \arg \max_{A \in \mathcal{A}} \langle \tilde{\theta}_A^{n+1}, s \rangle + \alpha [s^{\mathsf{T}}(W_A^{n+1})^{-1} s]^{\frac{1}{2}}$ , where  $\omega \geq 0$  and  $\alpha \geq 0$  are the regularization parameters in policy and their theoretical values are given in Theorem 4. We summarize the reward imputation using sketching and the sketched policy into Algorithm 2, called SPUIR. Similarly, we call the updating of the original policy that uses reward imputation without sketching, the Policy Updating with Imputed Rewards (PUIR).
<sup>3</sup>Since we set the number of blocks of SJLT as  $D < d$ , we omit  $D$  in the complexity analysis.
Table 1: The time complexities of the original reward imputation in (1) (first 4 columns) and the reward imputation using sketching in (5) (last 4 columns) for action  $A_{j}$  in the  $(n + 1)$ -th episode, where  $N_{j}^{n}\left(\widehat{N}_{j}^{n}\right)$  denotes the number of steps at which the action  $A_{j}$  is executed (non-executed) in episode  $n + 1$ ,  $\widehat{N}_j^n +N_j^n = B$ , and the sketch size  $c$  satisfying  $d < c < B$  and  $cd > B$  (MM: matrix multiplication; MI: matrix inversion; Overall: overall time complexity for action  $A_{j}$  in one episode)
Original reward imputation in (1)	Reward imputation using sketching in (5)
Item	Operation	Equation	Time	Item	Operation	Equation	Time
Φn+1Aj, Φn+1Aj	MM	(3), (4)	O(Bd2)	Gn+1Aj, Gn+1Aj	MM	(7), (8)	O(cd2)
bn+1Aj, bn+1Aj	MM	(3), (4)	O(Bd)	pnn+1Aj, pnn+1Aj	MM	(7), (8)	O(cd)
(Ψn+1Aj)-1	MI	(2)	O(d3)	(Wn+1Aj)-1	MI	(6)	O(d3)
-	ΓnAj, ΛnAj	Sketching	-	O(Nnj)d)
-	HnAj, ΛnAj	Sketching	-	O(Njnd)
Overall	-	-	O(Bd2)	Overall	-	-	O(cd2)
Algorithm 2 Sketched Policy Updating with Imputed Rewards (SPUIR) in the  $(n + 1)$ -th episode
INPUT: Policy  $\tilde{p}_n$ , data buffer  $\mathcal{D}_{n+1}$ ,  $\mathcal{A} = \{A_j\}_{j \in [M]}$ ,  $\alpha \geq 0$ ,  $\eta \in (0,1)$ ,  $\gamma \in [0,1]$ ,  $\lambda > 0$ ,  $\boldsymbol{W}_{A_j}^0 = \lambda \boldsymbol{I}_d$ ,  $\boldsymbol{G}_{A_j}^0 = \widehat{\boldsymbol{G}}_{A_j}^0 = \boldsymbol{O}_d$ ,  $\boldsymbol{p}_{A_j}^0 = \hat{\boldsymbol{p}}_{A_j}^0 = \boldsymbol{0}$ ,  $\tilde{\boldsymbol{\theta}}_{A_j}^0 = \boldsymbol{0}$ ,  $j \in [M]$ , batch size  $B$ , sketch size  $c$ , number of block  $D$   
OUTPUT: Updated policy  $\tilde{p}_{n+1}$
1: For all  $j \in [M]$ , store context vectors and rewards corresponding to the steps at which the action  $A_{j}$  is executed, into  $\Gamma_{A_j}^n \in \mathbb{R}^{N_j^n \times d}$  and  $\Lambda_{A_j}^n \in \mathbb{R}^{N_j^n}$
2: For all  $j \in [M]$ , store context vectors corresponding to the steps at which the action  $A_{j}$  is not executed into  $\hat{\Gamma}_{A_j}^n \in \mathbb{R}^{\hat{N}_j^n \times d}$ , where  $\hat{N}_j^n \gets B - N_j^n$
3:  $\tilde{r}_{n,b}(A_j) \leftarrow \langle \tilde{\theta}_{A_j}^n, s_{n,b} \rangle$ , for all  $A_j \in \mathcal{A}$  and  $b \in [\widehat{N}_j^n]$ , where  $s_{n,b}$  is the  $b$ -th row of  $\widehat{\Gamma}_{A_j}^n$
4: Compute imputed reward vector  $\widehat{\pmb{R}}_{A_j}^n\gets \{\tilde{r}_{n,1}(A_j),\dots ,\tilde{r}_{n,\widehat{N}_j^n}(A_j)\} \in \mathbb{R}^{\widehat{N}_j^n},\forall j\in [M]$
5: for all action  $A_{j}\in \mathcal{A}$  do
6:  $G_{A_j}^{n + 1}\gets G_{A_j}^n +\Gamma_{A_j}^{n\intercal}\Gamma_{A_j}^n,p_{A_j}^{n + 1}\gets p_{A_j}^n +\Gamma_{A_j}^{n\intercal}\Lambda_{A_j}^n$  {（7）}
$$
\widehat {\boldsymbol {G}} _ {A _ {j}} ^ {n + 1} \leftarrow \eta \widehat {\boldsymbol {G}} _ {A _ {j}} ^ {n} + \widehat {\Gamma} _ {A _ {j}} ^ {n \intercal} \widehat {\boldsymbol {\Gamma}} _ {A _ {j}} ^ {n}, \widehat {\boldsymbol {p}} _ {A _ {j}} ^ {n + 1} \leftarrow \eta \widehat {\boldsymbol {p}} _ {A _ {j}} ^ {n} + \widehat {\Gamma} _ {A _ {j}} ^ {n \intercal} \widehat {\boldsymbol {\Lambda}} _ {A _ {j}} ^ {n} \quad \{(8) \}
$$
7:  $\pmb{W}_{A_j}^{n + 1}\gets \lambda \pmb {I}_d + \pmb{G}_{A_j}^{n + 1} + \gamma \widehat{\pmb{G}}_{A_j}^{n + 1},\quad \tilde{\pmb{\theta}}_{A_j}^{n + 1}\gets (\pmb{W}_{A_j}^{n + 1})^{-1}(\pmb {p}_{A_j}^{n + 1} + \gamma \widehat{\pmb{p}}_{A_j}^{n + 1})\quad \{(6)\}$
8: end for
9:  $\tilde{p}_{n + 1}(\boldsymbol {s})$  selects action  $A\gets \arg \max_{A\in \mathcal{A}}\langle \tilde{\theta}_A^{n + 1},\boldsymbol {s}\rangle +\alpha [\boldsymbol{s}^{\intercal}\left(\boldsymbol{W}_A^{n + 1}\right)^{-1}\boldsymbol {s}]\frac{1}{2}$  for a new context  $\boldsymbol{s}$
10: Return  $\{\tilde{\theta}_A^{n + 1}\}_{A\in \mathcal{A}},\{\left(W_A^{n + 1}\right)^{-1}\}_{A\in \mathcal{A}}$
We provide the instantaneous regret bound, prove the approximation error of sketching, and analyze the regret of SPUIR in CBB setting. The detailed proofs can be found in Appendix B. We first demonstrate the instantaneous regret bound of the original solution  $\hat{\pmb{\theta}}_A^n$  in (1).
Theorem 2 (Instantaneous Regret Bound). Let  $\eta \in (0,1)$  be the discount parameter,  $\gamma \in [0,1]$  the imputation rate. In the  $n$ -th episode, if the rewards  $\{R_{n,b}\}_{b\in [B]}$  are independent and bounded by  $C_R$ , then, for any  $b\in [B],\forall A\in \mathcal{A},$  there exists  $C_\mathrm{Imp} > 0$  such that, with probability at least  $1 - \delta$
$$
\left| \left\langle \bar {\boldsymbol {\theta}} _ {A} ^ {n}, \boldsymbol {s} _ {n, b} \right\rangle - \left\langle \boldsymbol {\theta} _ {A} ^ {*}, \boldsymbol {s} _ {n, b} \right\rangle \right| \leq \left[ \lambda \| \boldsymbol {\theta} _ {A} ^ {*} \| _ {2} + \nu + \gamma^ {\frac {1}{2}} \eta^ {- \frac {1}{2}} C _ {\mathrm {I m p}} \right] \left[ \boldsymbol {s} _ {n, b} ^ {\intercal} \left(\boldsymbol {\Psi} _ {A} ^ {n}\right) ^ {- 1} \boldsymbol {s} _ {n, b} \right] ^ {\frac {1}{2}}, \tag {9}
$$
where  $\Psi_A^n = \lambda I_d + \Phi_A^n + \gamma \widehat{\Phi}_A^n$ ,  $\nu = [2C_R^2 \log (2MB / \delta)]^{\frac{1}{2}}$ . The first term on the right-hand side of (9) can be seen as the bias term for the reward imputation, while the second term is the variance term. The variance term of our algorithm is not larger than that without the reward imputation, i.e., for any  $s \in \mathbb{R}^d$ ,
$$
\left[ \boldsymbol {s} ^ {\intercal} \left(\boldsymbol {\Psi} _ {A} ^ {n}\right) ^ {- 1} \boldsymbol {s} \right] ^ {\frac {1}{2}} \leq \left[ \boldsymbol {s} ^ {\intercal} \left(\lambda \boldsymbol {I} _ {d} + \boldsymbol {\Phi} _ {A} ^ {n}\right) ^ {- 1} \boldsymbol {s} \right] ^ {\frac {1}{2}}.
$$
Further, a larger imputation rate  $\gamma$  leads to a smaller variance term  $\left[\pmb{s}^{\mathsf{T}}\left(\pmb{\Psi}_{A}^{n}\right)^{-1}\pmb{s}\right]^{\frac{1}{2}}$ .
<sup>4</sup>The assumption about conditional independence of the rewards is commonly used in the bandits literature, which can be ensured using a master technology as a theoretical construction (Auer, 2002; Chu et al., 2011).
Remark 1 (Smaller Variance). From Theorem 2, we can observe that our reward imputation achieves a smaller variance  $\left([s_{n,b}^{\mathsf{T}}(\Psi_A^n)^{-1}s_{n,b}]\frac{1}{2}\right)$  than that without the reward imputation. By combining Theorem 2 and the proof of Theorem 1, we can obtain that the variance in instantaneous regret bound of SPUIR is in between the variances in full and partial information scenarios. Thus, reward imputation in SPUIR provides a promising way to use expert advice approaches for bandit problems.
Remark 2 (Controllable Bias). Our reward imputation approach incurs a bias term  $\gamma^{\frac{1}{2}}\eta^{-\frac{1}{2}}C_{\mathrm{Imp}}$  in addition to the two bias terms  $\lambda \| \pmb{\theta}_A^*\| _2$  and  $\nu$  that exist in every existing UCB-based policy. But the additional bias term  $\gamma^{\frac{1}{2}}\eta^{-\frac{1}{2}}C_{\mathrm{Imp}}$  is controllable due to the presence of imputation rate  $\gamma$  that can help controlling the additional bias. Moreover, the term  $C_{\mathrm{Imp}}$  in the additional bias can be replaced by a function  $f_{\mathrm{Imp}}(n)$ , and  $f_{\mathrm{Imp}}(n)$  is monotonic decreasing w.r.t. number of episodes  $n$  provided that the mild condition  $\sqrt{\eta} = \Theta (d^{-1})$  holds (the definition and analysis about  $f_{\mathrm{Imp}}$  can be found in Appendix B.1). Overall, the imputation rate  $\gamma$  controls a trade-off between the bias term and the variance term, and we will design a rate-scheduled approach for automatically setting  $\gamma$  in Section 5.
Remark 3 (Relationship with Existing Instantaneous Regrets). According to the original definition in the context of online learning, the definition of instantaneous regret should be  $\max_{A\in \mathcal{A}}\langle \pmb{\theta}_{A}^{*},\pmb{s}_{n,b}\rangle -$ $\langle \pmb{\theta}_{A_{I_{n,b}}}^*$ $s_{n,b}\rangle$  . However, in the specific setting of contextual batchit (CBB) that is the focus of this paper, as derived in Appendix (second inequality of Eq. (48)), if we denote the upper bound of  $|\langle \bar{\pmb{\theta}}_A^n,s_{n,b}\rangle -\langle \pmb {\theta}_A^*,\pmb {s}_{n,b}\rangle |$  as  $U$  , then  $2U$  serves as an upper bound for instantaneous regret. Thus, in the context of CBB explored in this paper, we are interested in an upper bound for  $|\langle \bar{\pmb{\theta}}_A^n,s_{n,b}\rangle -\langle \pmb {\theta}_A^*,\pmb {s}_{n,b}\rangle |$  and define it as the instantaneous regret bound.
Although some approximation error bounds using SJLT have been proposed (Nelson and Nguyen, 2013; Kane and Nelson, 2014; Zhang and Liao, 2019), it is still unknown what is the lower bound of the sketch size while applying SJLT to the sketched ridge regression problem in our SPUIR. Next, we prove the approximation error as well as the lower bound of the sketch size in SPUIR. For convenience, we drop all the superscripts and subscripts in this result.
Theorem 3 (Approximation Error Bound of Imputation using Sketching). Denote the imputation regularized ridge regression function by  $F(\pmb{\theta})$  (defined in (1)) and the sketched ridge regression function by  $F^{\mathrm{S}}(\pmb{\theta})$  (defined in (5)) for reward imputation, whose solutions are  $\bar{\pmb{\theta}} = \arg \min_{\pmb{\theta} \in \mathbb{R}^d} F(\pmb{\theta})$  and  $\tilde{\pmb{\theta}} = \arg \min_{\pmb{\theta} \in \mathbb{R}^d} F^{\mathrm{S}}(\pmb{\theta})$ . Let  $\gamma \in [0,1]$  be the imputation rate,  $\lambda > 0$  the regularization parameter,  $\delta \in (0,0.1]$ ,  $\varepsilon \in (0,1)$ ,  $L_{\mathrm{all}} = [L; \sqrt{\gamma} \hat{L}]$ , and  $\rho_{\lambda} = \| L_{\mathrm{all}} \|_2^2 / (\| L_{\mathrm{all}} \|_2^2 + \lambda)$ . If  $\Pi$  and  $\widehat{\Pi}$  are SJLT, assuming that  $D = \Theta (\varepsilon^{-1} \log^3 (d\delta^{-1}))$  and the sketch size  $c = \Omega \left( d \operatorname{polylog}(d\delta^{-1}) / \varepsilon^2 \right)$ , with probability at least  $1 - \delta$ , the following results hold:
$$
F (\tilde {\boldsymbol {\theta}}) \leq (1 + \rho_ {\lambda} \varepsilon) F (\bar {\boldsymbol {\theta}}), \| \tilde {\boldsymbol {\theta}} - \bar {\boldsymbol {\theta}} \| _ {2} = O \left(\sqrt {\rho_ {\lambda} \varepsilon}\right).
$$
To measure the convergence of approximating the optimal policy in an online manner, we define the regret of SPUIR against the optimal policy as
$$
\operatorname {R e g} (N, B) := \max _ {A \in \mathcal {A}} \sum_ {n \in [ N ], b \in [ B ]} \left[ \langle \pmb {\theta} _ {A} ^ {*}, \pmb {s} _ {n, b} \rangle - \langle \pmb {\theta} _ {A _ {I _ {n, b}}} ^ {*}, \pmb {s} _ {n, b} \rangle \right],
$$
where  $I_{n,b}$  denotes the index of the executed action using the sketched policy  $\tilde{p}_n$  (parameterized by  $\{\tilde{\boldsymbol{\theta}}_A^n\}_{A\in \mathcal{A}}$ ) at step  $b$  in the  $n$ -th episode. We final prove the regret bound of SPUIR.
Theorem 4 (Regret Bound of SPUIR). Let  $T = BN$  be the overall number of steps,  $\eta \in (0,1)$  be the discount parameter,  $\gamma \in [0,1]$  the imputation rate,  $\lambda > 0$  the regularization parameter,  $C_{\theta^*}^{\max} = \max_{A \in \mathcal{A}} \| \pmb{\theta}_A^* \|_2$ ,  $C_{\mathrm{Imp}}$  be the positive constant defined in Theorem 2. Assume that the conditional independence assumption in Theorem 2 holds and the upper bound of rewards is  $C_R$ ,  $M = O(\mathrm{poly}(d))$ ,  $T \geq d^2$ ,  $\nu = [2C_R^2 \log(2MB / \delta_1)]^{\frac{1}{2}}$  with  $\delta_1 \in (0,1)$ ,  $\omega = \lambda C_{\pmb{\theta}^*}^{\max} + \nu + \gamma^{\frac{1}{2}} \eta^{-\frac{1}{2}} C_{\mathrm{Imp}}, \alpha = \omega C_\alpha$ , where  $C_\alpha > 0$  which decreases with increase of  $1 / \varepsilon$  and  $\varepsilon \in (0,1)$ . Let  $\delta_2 \in (0,0.1]$ ,  $\rho_\lambda < 1$  be the constant defined in Theorem 3, and  $C_{\mathrm{reg}}$  be a positive constant that decreases with increase of  $1 / \varepsilon$ . For the sketch matrices  $\{\Pi_A^n\}_{A \in A, n \in [N]}$  and  $\{\widehat{\Pi}\}_{A \in A, n \in [N]}$ , assuming that the number of blocks in SJLT  $D = \Theta (\varepsilon^{-1} \log^3(d\delta_2^{-1}))$ , and the sketch size satisfying
$$
c = \Omega \left(d \operatorname {p o l y l o g} \left(d \delta_ {2} ^ {- 1}\right) / \varepsilon^ {2}\right),
$$
then, for an arbitrary sequence of contexts  $\{s_{n,b}\}_{n\in [N],b\in [B]}$ , with probability at least  $1 - N(\delta_1 + \delta_2)$
$$
\operatorname {R e g} (N, B) \leq 2 \alpha C _ {\text {r e g}} \sqrt {1 0 M} \log (T + 1) (\sqrt {d T} + d B) + O \left(T \sqrt {\rho_ {\lambda} \epsilon d} / B\right). \tag {10}
$$
Remark 4. Setting  $B = O(\sqrt{T / d})$ ,  $\rho_{\lambda}\epsilon = 1 / d$  yields a sublinear regret bound of order  $\widetilde{O} (\sqrt{MdT})^5$  provided that the sketch size  $c = \Omega (\rho_{\lambda}^{2}d^{3}\log (d\delta_{2}^{-1}))$ . We can observe that the lower bound of  $c$  is independent of the overall number of steps  $T$ , and a theoretical value of the batch size is  $B = C_B\sqrt{T / d} = C_B^2 N / d$ , where setting  $C_B \approx 25$  is a suitable choice that has been verified in the experiments in Section 6. In particular, when  $\rho_{\lambda} = O(1 / d)$ , the sketch size of order  $c = \Omega (d\mathrm{polylog}d)$  is sufficient to achieve a sublinear regret.
From the theoretical results of regret, we can observe that our SPUIR admits several advantages: (a) The order of our regret bound (w.r.t. the overall number of steps) is not higher than those in the literature in the fully-online setting (Li et al., 2019; Dimakopoulou et al., 2019) that is a more simple setting than ours; (b) The first term in the regret bound (10) measures the performance of policy updating using imputed rewards (called "policy error"). From Theorem 2 and Remark 1&2, we obtain that, in each episode, our policy updating has a smaller variance than the policy without the reward imputation, and incurs a decreasing additional bias under mild conditions, leading to a tighter regret (i.e., smaller policy error) after some number of episodes. (c) The second term on the right-hand side of (10) is of order  $O(T\sqrt{\rho_{\lambda}\epsilon d} / B)$ , which is incurred by the sketching approximation using SJLT (called "sketching error"). This sketching error does not have any influence on the order of regret of SPUIR, which may even have a lower order with a suitable choice of  $\rho_{\lambda}\varepsilon$ , e.g., setting  $\rho_{\lambda}\varepsilon = T^{-1 / 4}d^{-1}$  yields a sketching error of order  $O(T^{3 / 8}d^{1 / 2})$  provided that  $c = \Omega (\rho_{\lambda}^{2}d^{3}\mathrm{polylog}(d\delta_{2}^{-1})\sqrt{T})$ .
At a fundamental level, the effectiveness of the proposed reward imputation can be attributed to the following two key factors:
(1) Leveraging contextual bandit structure: Traditional bandit methods only consider the structural assumptions for executed actions, leaving out non-executed ones. Our reward imputation approach incorporates a wide range of reward function structural assumptions, covering both executed and non-executed actions. By imputing missing rewards with observed data, we reduce the impact of missing data for a more accurate reward estimation.
(2) Balancing exploration and exploitation: Reward imputation's effectiveness arises from its impact on the exploration-exploitation trade-off. By incorporating imputed rewards, our proposed algorithms can make informed decisions even when observed rewards are incomplete. This enhances the agent's exploration strategy, helping it discover more valuable actions and reducing cumulative regret. Essentially, our reward computation approach approximates full-information feedback, mitigating the explore/exploit dilemma.
5 Extensions of Our Approach
To make the proposed reward imputation approach more feasible and practical, we tackle the following two research questions by designing variants of our approach following the theoretical results:
RQ1 (Parameter Selection): Can we set the imputation rate  $\gamma$  without tuning?
RQ2 (Nonlinear Reward): Can we apply the proposed reward imputation approach to the case where the expectation of true rewards is nonlinear?
Rate-Scheduled Approach. For RQ1, we equip PUIR and SPUIR with a rate-scheduled approach, called PUIR-RS and SPUIR-RS, respectively. From Remark 1&2, a larger imputation rate  $\gamma$  leads to a smaller variance while increasing the bias, while the bias term includes a monotonic decreasing function w.r.t. number of episodes under mild conditions. Therefore, we can gradually increase  $\gamma$  with the number of episodes, avoiding the large bias at the beginning of reward imputation. Specifically, we set  $\gamma = X\%$  for episodes from  $(X - 10)\% \times N$  to  $X\% \times N$ , where  $X \in [10, 100]$ .
Application to Nonlinear Rewards. For RQ2, we provide nonlinear versions of reward imputation. We use linearization technologies of nonlinear rewards, rather than directly setting the rewards as nonlinear functions (Valko et al., 2013; Chatterji et al., 2019), avoiding the linear regret or curse of
We use the notation of  $\widetilde{O}$  to suppress logarithmic factors in the overall number of steps  $T$ .
Table 2: Performance comparison of coupon recommendation on commercial product
Algorithm	CVR (mean ± std)	CTCVR (mean ± std)	Time (sec., mean ± std)
DFM-S	0.8656 ± 0.0473	0.3317 ± 0.0218	302.3140 ± 8.3045
SBUCB	0.8569 ± 0.0037	0.4277 ± 0.0084	43.5435 ± 0.3659
BEXP3	0.4846 ± 0.0205	0.2425 ± 0.0116	53.5001 ± 0.9220
BEXP3-IPW	0.4862 ± 0.0187	0.2436 ± 0.0113	56.0101 ± 1.4142
BLTS-B	0.8663 ± 0.0178	0.4285 ± 0.0157	218.2109 ± 1.8198
PUIR	0.8807 ± 0.0053	0.4411 ± 0.0029	184.3575 ± 2.2346
SPUIR	0.8770 ± 0.0059	0.4397 ± 0.0032	81.5753 ± 1.5879
PUIR-RS	0.8763 ± 0.0056	0.4389 ± 0.0030	180.4999 ± 1.7763
SPUIR-RS	0.8758 ± 0.0058	0.4391 ± 0.0031	80.8003 ± 2.9030
kernelization. Specifically, instead of using the linear imputed reward  $\tilde{r}_{n,b}(A_j) \coloneqq \langle \tilde{\pmb{\theta}}_{A_j}^n, \pmb{s}_{n,b} \rangle$ , we use the following linearized nonlinear imputed rewards, denotes by  $\mathcal{T}_{n,b}(\pmb{\theta}, A)$ :
(1) SPUIR-Exp. We assume that the expected reward is an exponential function as  $G_{\mathrm{E}}(\pmb{\theta}, \pmb{s}) = \exp(\pmb{\theta}^{\intercal}\pmb{s})$ . Then  $\mathcal{T}_{n,b}(\pmb{\theta}, A) = \langle \pmb{\theta}, \nabla_{\pmb{\theta}}G_{\mathrm{E}}(\pmb{\theta}, \pmb{s}_{n,b}) \rangle$ , where  $\nabla_{\pmb{\theta}}G_{\mathrm{E}}(\pmb{\theta}, \pmb{s}_{n,b}) = \exp(\pmb{\theta}^{\intercal}\pmb{s}_{n,b})\pmb{s}_{n,b}$ .
(2) SPUIR-Poly. When the expected reward is a polynomial function as  $G_{\mathrm{P}}(\pmb{\theta}, \pmb{s}) = (\pmb{\theta}^{\mathsf{T}}\pmb{s})^{2}$ . Then  $\mathcal{T}_{n,b}(\pmb{\theta}, A) = \langle \pmb{\theta}, \nabla_{\pmb{\theta}}G_{\mathrm{P}}(\pmb{\theta}, \pmb{s}_{n,b}) \rangle$ , where  $\nabla_{\pmb{\theta}}G_{\mathrm{P}}(\pmb{\theta}, \pmb{s}_{n,b}) = 2(\pmb{\theta}^{\mathsf{T}}\pmb{s}_{n,b})\pmb{s}_{n,b}$ .
(3) SPUIR-Kernel. Consider that the underlying expected reward in a Gaussian reproducing kernel Hilbert space (RKHS). We use  $\mathcal{T}_{n,b}(\pmb{\theta}, A) = \langle \pmb{\theta}, \phi(\pmb{s}_{n,b}) \rangle$  in random feature space, where the random feature mapping  $\phi$  can be explicitly computed.
For SPUIR-Exp and SPUIR-Poly, combining the linearization of convex functions (Shalev-Shwartz, 2011) with Theorem 4 yields the regret bounds of the same order. For SPUIR-Kernel, using the approximation error of random features (Rahimi and Recht, 2008), we can also obtain that, SPUIR-Kernel has the same regret bound as SPUIR under mild conditions (see proofs in Appendix B).
We empirically evaluated the performance of our algorithms on 3 datasets: the synthetic dataset, publicly available Criteo dataset $^6$  (Criteo-recent, Criteo-all), and dataset collected from Tencent's WeChat app for coupon recommendation (commercial product).
Experimental Settings. We compared our algorithms with: Sequential Batch UCB (SBUCB) (Han et al., 2020), Batched linear EXP3 (BEXP3) (Neu and Olkhovskaya, 2020), Batched linear EXP3 using Inverse Propensity Weighting (BEXP3-IPW) (Bistritz et al., 2019), Batched Balanced Linear Thompson Sampling (BLTS-B) (Dimakopoulou et al., 2019), and Sequential version of Delayed Feedback Model (DFM-S) (Chapelle, 2014). We applied the algorithms to CBB setting and implemented on Intel(R) Xeon(R) Silver 4114 CPU@2.20GHz, and repeated the experiments 20 times. We tested the performance of algorithms in streaming recommendation scenarios, where the reward is represented by a linear combination of the click and conversion behaviors of users. According to Remark 4, we set the batch size as  $B = C_B^2 N / d$ , the constant  $C_B \approx 25$ , and the sketch size  $c = 150$  on all the datasets. The average reward was used to evaluate the accuracy of algorithms.
Performance Evaluation. Figure 3(a)-(c) reports the average reward of SPUIR with its variants and the baselines. We observed that SPUIR and its variants achieved higher average rewards, demonstrating the effectiveness of our reward imputation. Moreover, SPUIR and its rate-scheduled version SPUIR-RS had similar performances compared with the origin PUIR, which indicates the practical effectiveness of our variants and verifies the correctness of the theoretical analyses. The results on commercial product in Table 2 indicate that SPUIR outperformed the second-best baseline with the improvements of  $1.07\%$  CVR (conversion rate) and  $1.12\%$  CTCVR (post-view
$^{6}$ https://labs.criteo.com/2013/12/conversion-logs-dataset/
(a) synthetic data
(b) Criteo-all
(c) commercial product
(d) nonlinear reward
(e) batch size  $B$
(f) sketch size  $c$
Figure 3: (a), (b), (c): Average rewards of the compared algorithms, the proposed SPUIR and its variants on synthetic dataset, Criteo dataset, and the real commercial product data, where we omitted the curves of algorithms whose average rewards are  $5\%$  lower than the highest reward; (d): SPUIR and its three nonlinear variants on synthetic dataset; (e): SPUIR with different batch sizes on Criteo-recent; (f): SPUIR and SPUIR-RS with different sketch sizes on synthetic dataset
click-through&conversion rate). Besides, our reward imputation approaches were more efficient than DFM-S, BLTS-B. The variants using sketching of our algorithms (SPUIR, SPUIR-RS) significantly reduced the time costs of reward imputation, and took less than twice as long to run compared to the baselines without reward imputation (SBUCB, BEXP3, BEXP3-IPW). Figure 3(d) illustrates performances of SPUIR and its nonlinear variants, where SPUIR-Kernel achieved the highest rewards indicating the effectiveness of the nonlinear generalization of our approach. For different decision tasks, a suitable nonlinear reward model needs to be selected for better performances.