SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} E _ {\pi} \left[ E \left[ \left(U (k, \mathbf {v}, q ^ {\prime}) - U \left(e _ {t, k} ^ {r}, \mathbf {v}, q ^ {\prime}\right)\right) 1 \left(\mathbf {v} _ {t} ^ {U C B} \geq \mathbf {v}\right) \mid \mathcal {F} _ {t - 1} \right] \right] \\ \leq E _ {\pi} \left[ E \left[ \left(v _ {e _ {t, k} ^ {r}, t} ^ {U C B} - v _ {e _ {t, k} ^ {r}}\right) 1 (\mathbf {v} _ {t} ^ {U C B} \geq \mathbf {v}) | \mathcal {F} _ {t - 1} \right] \right], \\ \end{array}
$$
where  $e_{t,k}^{r}$  is the  $k^{th}$  message sent to user  $r$  at time  $t$ .
Theorem 3.4 (Performance bound for Algorithm 2). The expected regret of Algorithm 2 is bounded above by
$$
R e g (T) \leq C _ {1} (N + M ^ {2}) \sqrt {T \log T} + C _ {2} N \tau_ {\max }
$$
for some constants  $C_1$  and  $C_2$ .
Proof Outline: First we note that
$$
\begin{array}{l} E _ {\pi} \left[ U \left(\mathbf {S} ^ {*}, \mathbf {v}, \mathbf {R}, q \left(m ^ {*}\right)\right) \right] - E _ {\pi} \left[ U \left(\mathbf {S} ^ {r}, \mathbf {v}, \mathbf {R}, q \left(m _ {r}\right)\right) \right] \\ = E _ {\pi} \left[ U \left(\mathbf {S} ^ {*}, \mathbf {v}, \mathbf {R}, q \left(m ^ {*}\right)\right) - U \left(\mathbf {S} ^ {*}, \mathbf {v}, \mathbf {R}, q \left(m _ {r}\right)\right) \right] \\ + E _ {\pi} \left[ U \left(\mathbf {S} ^ {*}, \mathbf {v}, \mathbf {R}, q \left(m _ {r}\right)\right) - U \left(\mathbf {S} ^ {r}, \mathbf {v}, \mathbf {R}, q \left(m _ {r}\right)\right) \right] \tag {3.2} \\ \end{array}
$$
Let  $\mathbf{S}_0^r$  denote the initial list for user  $r$  when she just arrives, i.e.,  $\mathbf{S}_0^r$  is the optimal sequence given  $\mathbf{v}_{r - 1}^{UCB}$  and  $\mathbf{q}_{r - 1}^{UCB}$ . Note that this list may change at a later time when more information becomes available. Define events  $B_{i,t} = \{v_{i,t}^{UCB} - \sqrt{8\log t / T_i(t)} < v_i < v_{i,t}^{UCB}\}$  and  $E_{m,t} = \{q_t^{UCB}(m) - \sqrt{8\log t / \hat{T}_m(t)} < q(m) < q_t^{UCB}(m)\}$ . Define  $H_{t} = \bigcap_{i\in X}B_{i,t}\bigcap_{1\leq m\leq M}E_{m,t}$ . Firstly we could estimate the upper bound of  $E_{\pi}[U(\mathbf{S}^{*},\mathbf{v},\mathbf{R},q(m^{*})) - U(\mathbf{S}^{*},\mathbf{v},\mathbf{R},q(m_r))]$  by coupling. We can get
$$
\begin{array}{l} \sum_ {r = 1} ^ {T} E _ {\pi} \left[ E \left[ \left(U \left(\mathbf {S} _ {0} ^ {r}, \mathbf {v} _ {r - 1} ^ {U C B}, \mathbf {R}, q _ {r - 1} ^ {U C B} (m _ {r})\right) - U \left(\mathbf {S} _ {0} ^ {r}, \mathbf {v}, \mathbf {R}, q (m _ {r})\right)\right) 1 \left(H _ {r - 1}\right) \mid \mathcal {F} _ {r - 1} \right] \right] \\ \leq C _ {2} N \sqrt {T \log T} + C _ {3} M ^ {2} \sqrt {T \log T}. \\ \end{array}
$$
Regarding the second part in Equation (3.2), we now bound the difference between  $E_{\pi}[U(\mathbf{S}^{*}, \mathbf{v}, \mathbf{R}, q(m_{r}))]$  and  $E_{\pi}[U(\mathbf{S}^{r}, \mathbf{v}, \mathbf{R}, q(m_{r}))]$ . Note that  $\mathbf{S}^r$  is an updated list that may differ from  $\mathbf{S}_0^r$ , so we again use coupling to bound the difference. We couple the processes sending  $\mathbf{S}^*$  and  $\mathbf{S}^r$ , which both have the identical dissemination frequency  $m_{r}$ .
Summing over all time steps, we have  $\sum_{r=1}^{T} E_{\pi}[U(\mathbf{S}^*, \mathbf{v}, \mathbf{R}, q(m_r)) - U(\mathbf{S}^r, \mathbf{v}, \mathbf{R}, q(m_r))] \leq C_4 \sqrt{\log T} \sum_{i=1}^{N} E_{\pi}\left[\sum_{t=1}^{T} z_{i,t} \sqrt{\frac{1}{T_i(t-1)}}\right] + DE_{\pi}\left[\sum_{t=1}^{T} 1(J_t^c)\right]$  where  $J_t = \bigcap_{i \in X} B_{i,t}$ . With some derivations, for each  $i \in X$ , we can bound the term
$$
E _ {\pi} \left[ \sum_ {t = 1} ^ {T} z _ {i, t} \sqrt {\frac {1}{T _ {i} (t - 1)}} \right] \leq C _ {5} \tau_ {m a x} + C _ {6} \sqrt {T}.
$$
Applying Lemma 3.1, we have
$$
E _ {\pi} \left[ \sum_ {t = 1} ^ {T} 1 \left(H _ {t} ^ {c}\right) \right] \leq C _ {7} (N + M).
$$
Combining all the results above, we arrive at the desired result.
The complete proofs of Lemma 3.2, Lemma 3.3, and Theorem 3.4 are included in the supplementary material. If the delay effect is absent, we can show that the regret bound in Theorem 3.4 is reduced to  $C_1(N + M^2)\sqrt{T\log T}$ . In other words, we show that the increase in the regret contributed by the delayed feedback is additive and linear in  $\tau_{max}$  in our setting. In this paper, we mainly focus on the instance-independent regret, while it is possible to derive an instance-dependent result similarly following Theorem 2 and Theorem 4 in [13], combined with the technique that we have used here to deal with multiple interactions and the delayed feedback.
As a special case of our proposed model, we can compare this result with the state-of-the-art results of cascading bandits. [15] has derived lower bound as  $\Omega (\sqrt{NMT})$ . Moreover, [11] has shown the regret of  $O(\sqrt{NT}\log T + NE[\tau])$ . Our setup is more complicated since there are more unknown parameters that need to be learned (probability of abandonment), and the feedback is received in a delayed manner. The regret bound for our proposed algorithm is  $O(\max (N,M^2)\sqrt{T} +N\tau)$ . From the comparison to two special cases, we can see that regret is optimal in  $T$  as well as the delay duration.
4 Personalized Recommendation
In the previous section, we assume that all users share the identical preferences towards content and the abandonment behavior. To provide personalized recommendations which incorporate content and
user features, we propose a contextual variant of the learning algorithm which is motivated by [17]. Note that in this section, we allow the message set  $X$  to vary for different users, denoted as  $X_{r}$ .
Define  $\mathbf{w}_{r,i}$  as the features of message  $i$  in  $X_{r}$ , and  $\mathbf{x}_r$  as user's features. We assume the attraction probability of message  $i$  is a logit function of its content feature  $\mathbf{w}_{r,i}$ , and user  $r$ 's abandonment behavior can be described by a logit function of the user's features  $\mathbf{x}_r$ . Define the link function  $\mu(y) = \exp(y) / (1 + \exp(y))$ . That is,
$$
1 - P (\text {a b a n d o n} | \mathbf {x} _ {r}, m) = \exp \left(\mathbf {x} _ {r} ^ {\prime} \alpha_ {m}\right) / \left(1 + \exp \left(\mathbf {x} _ {r} ^ {\prime} \alpha_ {m}\right)\right) = \mu \left(\mathbf {x} _ {r} ^ {\prime} \alpha_ {m}\right),
$$
$$
P (\text {c l i c k} i | \mathbf {w} _ {r, i}) = \exp \left(\mathbf {w} _ {r, i} ^ {\prime} \beta\right) / \left(1 + \exp \left(\mathbf {w} _ {r, i} ^ {\prime} \beta\right)\right) = \mu \left(\mathbf {w} _ {r, i} ^ {\prime} \beta\right).
$$
Set  $\hat{Y}_{r,j} = 1$  if user  $r$  remains in the system after she does not click on the  $j^{th}$  message in a list, while  $\hat{Y}_{r,j} = 0$  if she abandons the system after viewing the  $j^{th}$  message. Therefore, the maximum likelihood estimator of  $\alpha_{m}$  at time  $t$ ,  $\hat{\alpha}_{m,t}$ , can be obtained as
$$
\hat {\alpha} _ {m, t} = \arg \max _ {\boldsymbol {\alpha}} \sum_ {r = 1} ^ {t} \sum_ {j = 1} ^ {m _ {r}} 1 (m _ {r} = m) \left(\hat {Y} _ {r, j} \mathbf {x} _ {r} ^ {\prime} \boldsymbol {\alpha} - \log (1 + e ^ {\mathbf {x} _ {r} ^ {\prime} \boldsymbol {\alpha}})\right)
$$
$$
1 (\text {u s e r} r \text {e x a m i n e s t h e} j ^ {t h} \text {m e s g a g e b u t d o e s n o t c l i c k b y t i m e} t). \tag {4.1}
$$
Similarly, let  $Y_{r,j} = 1$  if user  $r$  clicks on the message  $j$ , and  $Y_{r,i} = 0$  otherwise. The maximum likelihood estimator of  $\beta$  at time  $t$  can be obtained as
$$
\hat {\beta} _ {t} = \arg \max  _ {\beta} \sum_ {r = 1} ^ {t} \sum_ {j = 1} ^ {m _ {r}} \left(Y _ {r, j} \mathbf {w} _ {r, \kappa_ {r} (j)} ^ {\prime} \beta - \log \left(1 + e ^ {\mathbf {w} _ {r, \kappa_ {r} (j)} ^ {\prime} \beta}\right)\right)
$$
$$
1 (\text {u s e r} r \text {g i v e s t h e f e a b d k f o r t h e} j ^ {t h} \text {m e s s a g e b y t i m e} t), \tag {4.2}
$$
where feedback includes both clicks and no-clicks and  $\kappa_{r}$  is the index function for user  $r$ .
Define the covariance matrices at time  $t$  for two estimators as  $M_{m,t} = \sum_{r=1}^{t} \sum_{j=1}^{m_r} \mathbf{x}_r \mathbf{x}_r' 1$  (user  $r$  examines the  $j^{th}$  message but does no click by time  $t$ )  $1(m_r = m)$ , and  $V_t = \sum_{r=1}^{t} \sum_{j=1}^{m_r} \mathbf{w}_{r,\kappa(j)} \mathbf{w}_{r,\kappa(j)}' 1$  (user  $r$  gives the feedback for the  $j^{th}$  message by time  $t$ ). We propose the following contextual learning algorithm. In each step, after observing users' feedback, we update the maximum likelihood estimators for  $\alpha_m$  and  $\beta$ , respectively, followed by their upper confidence bound. We then use these estimators to determine the messages for the active users.
Algorithm 3: An online algorithm for contextual cascading bandits with delayed feedback
1 Input: Total time steps  $T$ , tuning parameter  $\eta$ ,  $\gamma_{1}$  and  $\gamma_{2}$ ;  $t = 0$ ;  
2 Initialize: Randomly choose frequency and messages to send until time  $\eta$ . Update  $V_{\eta}$  and  $M_{m,\eta}$ ;  
3 while  $t < T$  do  
4 Update  $\hat{\alpha}_{m,t}$  and  $\hat{\beta}_t$  by Equation (4.1) and Equation (4.2);  $t = t + 1$ ;  
5 Observe a user's contexts  $\mathbf{x}_t$  and message contexts  $\mathbf{w}_{t,i}$ ; Update  
[ v_{t - 1}^{UCB}(\mathbf{w}_{t,i}) = \mu \left(\mathbf{w}_{t,i}'\hat{\beta}_{t - 1} + \gamma_1\| \mathbf{w}_{t,i}\|_{V_{t - 1}^{-1}}\right) ] for all  $i$ , and  
[ q_{m,t - 1}^{UCB}(\mathbf{x}_t) = \mu \left(\mathbf{x}_t'\hat{\alpha}_{m,t - 1} + \gamma_2\| \mathbf{x}_t\|_{M_{m,t - 1}^{-1}}\right) ] for all  $m$   
for Any active user  $r$  with message scheduled to be sent at time  $t$  do  
6 Compute  $\mathbf{S}^r = \arg \max_{\mathbf{S}} E[U(\mathbf{S},\mathbf{v}_{t - 1}^{UCB},\mathbf{R},q_{t - 1}^{UCB}(m)]$  according to Algorithm 1;  
7 Send the first message  $e$  in  $\mathbf{S}^r\backslash O^r$  to user  $r$ ;  $O_r = O_r\cup \{e\}$ ;  
8 end  
9 Compute  $(\mathbf{S}^t,m_t) = \arg \max_{s,m}E[U(\mathbf{S},\mathbf{v}_{t - 1}^{UCB},\mathbf{R},q_{t - 1}^{UCB}(m)]$  according to Algorithm 1;  
10 Offer  $S_1^t$  to user  $t$  and the dissemination frequency is set to  $f(m_t)$ ;  $O_t = S_1^t$ ;  
11 Update  $M_{m,t}$  and  $V_t$ ;  
12 end
It is also important to note that although Algorithm 2 and Algorithm 3 are both based on the assumption that messages cannot be recommended repeatedly, they can be easily generalized. If
$O_{r}$  is not excluded from the candidate messages, the two algorithms can be applied the repeat-recommendation case. Moreover, if we want to constrain the total number of repetitive displays of one message as  $B$ , we can create  $B$  copies of that message in the candidate set.
In this section, we evaluate the performance of both the non-contextual and contextual algorithms based on the features of the real AmEx User Click dataset<sup>3</sup>, which records over 463,000 recommendations of AmEx from July 2 to July 7 in 2017. The combination of "product" and "campaign_id" is regarded as messages in simulation.
Experiment I: Non-contextual setting. There are  $N = 25$  available messages. The attraction probability  $v$  is uniformly generated from distribution [0, 0.5], and the return  $R$  uniformly generated from distribution [1, 3]. The maximum length of message list  $M$  is set to be 10. Based on estimations from the short lists in AmEx dataset, we set  $q(m) = \frac{1}{1 + \exp(0.03m)}$ . We set the re-targeting window  $D = 200$  in all settings. The response time  $\tau$  is uniformly generated from [0, 3] for each user. In addition to our proposed algorithm, we introduce several algorithms as benchmarks: three different  $\epsilon$ -greedy algorithms and TS-Cascade, which is a Thompson sampling algorithm for cascading bandits with Gaussian update [25].
Result: We observe the results of 100 independent simulations for  $T = 30,000$  time steps. The results are shown in Fig 2, where the shaded area represents the  $95\%$  confidence region. We find that the optimistic greedy and decaying  $\epsilon$ -greedy algorithms are difficult to find an optimal list. The average regret at  $T = 30,000$  of our UCB algorithm is 550.14, with a  $95\%$  confidence interval of (454.71, 645.57), significantly better than the benchmarks.
Figure 2: Regrets of Algorithm 2.
Figure 3: Regrets of Algorithm 3.
Experiment II: Contextual setting. There are  $N = 25$  available messages. The maximum length of message list  $M$  is set to be 20. User features are uniformly generated from  $[0,5]\times [0,5]\times [0,5]$ , and message features are uniformly generated from  $[-6,0]\times [0,1]\times [0,2]\times [-5,0]$ . The coefficient related to the abandonment behavior, denoted as  $\alpha_{m}$ , is uniformly generated in the range (four-dimensional including the intercept)  $-1.04\times [-0.064m,0]\times [-0.08m,0]\times [-0.16m,0]$  for  $m = 1,\dots ,20$ , where  $\alpha_{m,1}$  is the intercept. An alternative coefficient  $\tilde{\alpha}_{m}$  is uniformly generated from  $-1.04\times [-0.004m,0]\times [-0.064m,0]\times [-0.08m,0]$  for  $m = 1,\dots ,20$ . The coefficient related to message attraction is  $\beta = (0.05,0.2,0.1,0.3,0.4)$ , where  $\beta_{1}$  is the intercept. Users' response time is uniformly distributed on [0,10]. We also implement several benchmark algorithms: two variants of the  $\epsilon$ -greedy algorithm, the explore-then-commit algorithm, and LinTS-Cascade, which is a linear generalization of TS-Cascade [25].
Result: We observe the results of 50 independent simulations for  $T = 3,000$  time steps. The result is shown in Fig 3. The average regret at  $T = 3,000$  of Algorithm 3 is 293.94, with a  $95\%$  confidence interval of (273.85, 314.04), significantly better than the benchmark algorithms. In the early stage,
<sup>3</sup>https://www.kaggle.com/code/muditagraphal/amex-user-click-prediction
the contextual UCB algorithm also has good performance, showing a faster convergence speed than benchmarks.
Experiment III: Comparison of UCB and contextual UCB. To compare the effect of Algorithm 2 and Algorithm 3, we make two settings: (a):  $N = 25$ ,  $M = 20$  with user features and message features generated from the same distribution as in Experiment 2, but only generated once and then fixed. All other parameters are the same as in Experiment 2; (b): same as (a) but with  $N = 100$ .
Figure 4: Comparison of the two UCB algorithms.
Result: We observe the results of 50 independent simulations for  $T = 3,000$  time steps. The result is shown in Fig 4. The average regrets under the 4 settings are 1451.95, 798.94, 441.90 and 280.73 from high to low, respectively. The contextual bandit UCB always work better than UCB algorithm in the same setting. Furthermore, the advantage of contextual algorithm becomes greater when there are more messages. This is because more messages will improve the estimation accuracy of  $\alpha$  and  $\beta$ .
In this work, we considered a cascading bandit problem with delayed feedback and flexible rewards. Based on the users' feedback, the learning agent needs a policy to select a list of messages and determine the frequency at which these messages are disseminated. We proposed a novel learning algorithm to effectively learn such a policy, and it is capable of recommending a list that is dynamic to a given user within polynomial time. Experiments proved the effectiveness and applicability of our algorithm compared to benchmark algorithms in a novel cascading bandit problem.
For future work, it would be interesting to analyze the performance of alternative bandit algorithms, such as Thompson-sampling-based algorithms for this setting. Additionally, considering that users' preferences may change over time, another avenue for future research would be to incorporate non-stationarity into user preferences.
Dairui Wang, Yan Zhang, and Wei Qi acknowledge the support from National Natural Science Foundation of China (Grants 72242106, 72188101).
[1] Alekh Agarwal and John C Duchi. Distributed delayed stochastic optimization. In Advances in Neural Information Processing Systems, pages 873-881, 2011.
[2] Junyu Cao and Wei Sun. Dynamic learning of sequential choice bandit problem under marketing fatigue. In The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), 2019.
[3] Nicolo Cesa-Bianchi and Gabor Lugosi. Combinatorial bandits. Journal of Computer and System Sciences, 78(5):1404-1422, 2012.
[22] Marcelo J Weinberger and Erik Ordentlich. On delayed prediction of individual sequences. IEEE Transactions on Information Theory, 48(7):1959-1976, 2002.
[23] Jiaqi Yang and De-Chuan Zhan. Generalized delayed feedback model with post-click information in recommender systems. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 26192-26203. Curran Associates, Inc., 2022.
[24] Yiping Yuan, Ajith Muralidharan, Preetam Nandy, Miao Cheng, and Prakruthi Prabhakar. Offline Reinforcement Learning for Mobile Notifications. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 3614-3623, Atlanta GA USA, October 2022. ACM.
[25] Zixin Zhong, Wang Chi Chueng, and Vincent YF Tan. Thompson sampling algorithms for cascading bandits. The Journal of Machine Learning Research, 22(1):9915-9980, 2021.