SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dairui Wang  
Tsinghua University  
wdr23@mails.tsinghua.edu.cn
Yan Zhang  
McGill University  
yan.zhang13@mail.mcgill.ca
Junyu Cao*  
The University of Texas at Austin  
junyu.cao@mccombs.utexas.edu
Wei Qi* Tsinghua University qiw@tsinghua.edu.cn
Delayed feedback is a critical problem in dynamic recommender systems. In practice, the feedback result often depends on the frequency of recommendation. Most existing online learning literature fails to consider optimization of the recommendation frequency, and regards the reward from each successfully recommended message as equal. In this paper, we consider a novel cascading bandits setting, where individual messages from a selected list are sent to a user periodically. Whenever a user does not like a message, she may abandon the system with a probability positively correlated with the recommendation frequency. A learning agent needs to learn both the underlying message attraction probabilities and users' abandonment probabilities through the randomly delayed feedback. We first show a dynamic programming solution to finding the optimal message sequence in deterministic scenarios, in which the reward is allowed to vary with different messages. Then we propose a polynomial time UCB-based offline learning algorithm, and discuss its performance by characterizing its regret bound. For the online setting, we propose a learning algorithm which allows adaptive content for a given user. Numerical experiment on AmEx dataset confirms the effectiveness of our algorithms.
1 Background
1.1 Introduction
Recommending dynamic content based on adaptive learning of user behavior is a critical task to online platforms. In marketing campaign, as only  $2\%$  of web traffic converts on the first visit<sup>1</sup>, it is important to re-target potential users with notifications such as ads, emails or app reminders to gain traction and awareness. The content for those notifications should be dynamic and adapted to a user's behavior. Since the feedback may not be instant, a campaign system still needs to provide campaigns to other users while waiting. Meanwhile, the frequency at which the notifications are sent should also be judiciously managed, i.e., sending more messages may increase the odds of a click, but frequent messaging may also alienate and drive away users.
Online learning for recommendation tasks has been extensively studied in the multi-armed bandit framework. In this setting, pulling certain arms in the bandit problem corresponds to selecting marketing messages for a user. In particular, cascading bandit [13, 14, 18, 21, 20, 25] which models
\*Corresponding authors. 1https://retargeter.com/what-is-retargeting-and-how-does-it-work/
sending a sequence of messages to individual users is a popular choice for analyzing sequential recommendations. However, existing settings in this field are often stylized and overlook certain characteristics that are prevalent in real-world scenarios. For instance, a common assumption is immediate user feedback, whereas in practice, it may take hours to months to receive responses. Additionally, bandit models typically disregard the variation in payback for each message, which limits the practical implications of the model for recommendations. Furthermore, the concept of "frequency" in message dissemination is absent in current cascading bandit literature, where an entire sequence of messages is sent to a user at once. Consequently, the content for a single user becomes non-adaptive, potentially missing out on additional feedback from other users over an extended time span and foregoing potential benefits.
In this work, we propose and analyze a variant of cascading bandit with five notable differences from the existing literature. Firstly, whenever a user does not like a message, she may abandon the system and forgo receipt of future messages. Thus, the order of messages influences the reward in our setting, which is in contrast to the existing work on cascading bandit such as [13, 14, 18, 21, 20, 25]. Secondly, instead of having the total number of messages in a sequence as a fixed parameter, this quantity is also a decision variable. In particular, we assume the abandonment probability may increase with the frequency at which messages are being disseminated. Thirdly, individual messages for each user are sent out with some time apart, as opposed to all at once. As we gather more feedback from some of the previous messages, we have the ability to update the remaining sequence, and thus the content for a user is dynamic and adaptive. Fourthly, the reward generated after a message is clicked varies with the type of it. Lastly, we also incorporate delayed feedback in our setting.
The analysis for this setting is particularly challenging. One source of complications comes from the interlaced timing of multiple events, i.e., users arrive at different time, messages for individual users are sent at different time, and their responses time is also random. Meanwhile, users' feedback also depends on multiple factors, including the choices of the content, their order, as well as the dissemination frequency.
The contribution of our work is fourfold. 1. We introduce a novel cascading bandit setting with delayed feedback, different reward and frequency control. 2. For the offline combinatorial problem with all known parameters, we characterize a polynomial-time algorithm which determines an optimal list with the appropriate dissemination frequency. 3. For the online setting where we need to learn both the message attraction probabilities and the abandonment probabilities, we propose a learning algorithm and characterized its regret bound. 4. We incorporate personalization by analyzing a contextual variant of the online problem which utilizes both content and user features.
The paper is organized as follows. We review the related literature in Section 1.2. In Section 2, we formally introduce the model and analyze the offline optimization problem with all known parameters. In Section 3 and Section 4, we study the online learning problem in the presence of delayed feedback for the non-contextual and contextual settings, respectively. We evaluate the performance of our algorithms with numerical experiments in Section 5. Lastly, we discuss some limitations and research prospect of our work in Section 6.
The analysis on delayed feedback has started to receive increasing attention in recent years [1, 22, 16, 23]. In online learning, [8] considers a fixed and known delay effect, and shows an additive penalty of the regret for the stochastic setting. [11] analyzes a more general setting and show that the regret increase is multiplicative in the adversarial setting and additive in the stochastic setting. They propose meta-algorithms that can adapt non-delay algorithms and develop a UCB algorithm for the bandit problem. [7] and [19] analyze the impact of delayed feedback in a Gaussian process bandit and an online Markov decision process respectively. Compared to the existing bandit literature, to the best of our knowledge, our work is the first to incorporate delay effects for the cascade model which involves a sequence of actions. Besides the uncertainty contributed by the random response time, the feedback also depends on both the choice of the messages and the order in which they are being sent.
Our work shares some similarity with combinatorial bandits [3, 5, 15, 4] as the learning agent needs to choose a set of messages. However, in our setting, the order of the messages also impacts the rewards. As highlighted in the introduction, our work extends the cascading bandits literature by including frequency control and delayed feedback. This setting can be potentially modeled as a reinforcement
learning problem, such as [24], which proposes an offline reinforcement learning framework to optimize sequential notification decisions. However, this approach significantly increases the learning complexity, making it difficult to analyze performance theoretically.
In the original cascading model [6], a user who is presented with a list of messages examines the content sequentially, until she clicks on the first attractive message. A common different assumption is the dependent click model. In this model, a user may click on multiple messages and finally stop at the truly attractive one. It can be used in bandit settings [12] or generic click-models algorithms [9]. In cascading bandit [13, 14, 18, 21, 20, 25], the attraction probabilities of messages are not known a priori. A learning agent selects a list with a fixed number of messages to incoming users, with the goal to maximize the total clicks over  $T$  time steps. User's feedback to a list is immediate. The agent then uses this information to learn the underlying attraction probabilities and updates their list for the next user. Although [10] provides a bound based on Thompson sampling for semi-bandit rewards and [20] provides a problem-independent bound for unstructured rewards, they do not consider the messages' difference in theory. While [2] also considers user abandonment behavior in a cascading bandit setting, the abandonment probability is a constant and the recommendation to a single user is static, as opposed an adaptive sequence which is considered in this work.
Assume there are  $N$  available messages, which we denote as  $X$ . We use  $\mathbf{S} = (S_1, S_2, \dots, S_m)$  to denote a list of messages, where  $S_1$  and  $S_m$  represent the first and the last message, respectively. Let  $\kappa$  be the index function mapping the position in a list to a message, i.e.,  $\kappa(i) = j$  if and only if  $S_i = \{j\}$ . Define  $\vartheta$  as the inverse function of  $\kappa$ , i.e.,  $\vartheta(i) = j$  if and only if  $S_j = \{i\}$ .
Users will receive a list of messages periodically. Upon receiving a message, if a user finds it attractive, she clicks on it. Otherwise, she may examine the next message with probability  $q(m)$  if the list has not run out, or she may leave with probability  $1 - q(m)$ . The probability  $q(m)$  depends on the total length of the list or the message dissemination frequency, which we will elaborate next. For each message  $i$ , we use  $v_{i}$  to denote its attraction probability which captures the relevance of its content to a user, and use  $R_{i}$  to denote its return when a user clicks on it. We define  $\mathbf{v} = (v_{1},\dots ,v_{N})$  and  $\mathbf{R} = (R_1,\dots ,R_N)$ .
A user arrives at time  $t$ , where  $t = 1, \dots, T$ . We assume that for every user, there is a fixed time window  $D$ , during which selected messages will be disseminated<sup>2</sup>. Messages are sent sequentially at a fixed time interval with a dissemination frequency  $f(m) := \lceil m / D \rceil$ . Here we keep the recommendation frequency fixed for every user, which allows us to easily capture the trade-off between the total number of messages that can be disseminated and the probability of abandonment. At every time step, a user receives at most one message. Define  $M$  as the maximum number of messages that can be sent during  $D$ . We assume  $q(m)$  is a non-increasing function with  $m$ . The intuition is that when the frequency of re-targeting is higher, a user is more likely to be overwhelmed and is more prone to abandonment. We denote  $\mathbf{q} = (q(1), \dots, q(M))$ . The following example illustrates the interactions between users and messages at different time.
Example 2.1. Figure 1 shows 5 users that arrive at time  $t = 1, \dots, 5$ . Actions taken by users are color-coded, i.e., a user clicks on a specific message (orange), has not clicked but remains active (green), and has abandoned the system (yellow).  $M_{i}, t = j$  denotes that message  $i$  is sent at time step  $j$ , e.g., User 2 receives message 10 at  $t = 2$  and message 9 at  $t = 6$ . While the dissemination frequency is determined for a given user upon arrival, it may be different across users, e.g., User 2 and 5 receive messages at a fixed interval of 4 and 6 time steps respectively.
For a list with  $m$  messages, besides the first message which is guaranteed to be viewed, the probability that message  $i$  is being examined, denoted as  $w_{i}(m)$ , depends on the outcomes of the earlier messages, i.e.,  $w_{i}(m) = \prod_{j=1}^{\vartheta(i)-1}((1 - v_{\kappa(j)})q(m))$ . It is the probability that the user does not click on any messages shown before  $i$  and still remains in the system. Let  $U(\mathbf{S},\mathbf{v},q(m))$  denote the payoff that the learning agent receives from a list of messages  $\mathbf{S}$ , when the message attraction probabilities are  $\mathbf{v}$
${}^{2}D$  is known as the re-targeting window in the marketing literature. See https://www.adroll.com/ learn-more/retargeting.
Figure 1: An illustrative example on user's interactions with the learning agent.
and abandonment probability is  $1 - q(m)$ . The expected payoff is
$$
E [ U (\mathbf {S}, \mathbf {v}, \mathbf {R}, q (m)) ] = \sum_ {i \in X} w _ {i} (m) R _ {i} v _ {i}.
$$
To determine a list of messages with appropriate dissemination frequency, the learning agent needs to solve an optimization problem that maximizes  $E[U(\mathbf{S},\mathbf{v},\mathbf{R},q(m))]$ , subject to a constraint that  $S_{i}\cap S_{j} = \emptyset$  for  $i\neq j$ , indicating no duplicated messages in the list. We use  $\mathbf{S}^*$  to denote the optimal solution to the combinatorial optimization problem.
In the following result, we characterize the property of the optimal list  $\mathbf{S}^*$ . We then propose an algorithm to identify  $\mathbf{S}^*$  in the offline setting, i.e., when the message attraction probabilities  $\mathbf{v}$  and the probabilities  $\mathbf{q}$  describing users' abandonment behavior are known.
Theorem 2.2. In the optimal sequence  $\mathbf{S}^*$ , the characteristic parameter of messages
$$
\gamma = \frac {v R}{1 - v (1 - q (m))}
$$
are sorted in a descending order.
Proof outline: We prove the statement by contradiction. If  $\mathbf{S}$  is not ordered by  $\gamma$ , we generate another sequence  $\mathbf{S}'$  by swapping a pair of adjacent messages in  $\mathbf{S}$ . We can prove that the expected payoff of  $\mathbf{S}'$  is higher than  $\mathbf{S}$ , which is a contradiction.
We include complete proof in the supplementary material. Note that while Theorem 2.2 provides a necessary condition for  $\mathbf{S}^*$ , it does not indicate that the messages with the highest  $\gamma$  should be selected. The structure of  $E[U(\mathbf{S},\mathbf{v},\mathbf{R},q(m))]$  shows that each message in the list will only influence the contribution of all the messages after it on the total return. Therefore, we use a dynamic programming approach to determine the list backwards.
Algorithm 1: Determine the optimal sequence  $\mathbf{S}^*$  with frequency control.
1 Input messages and their  $v,R$    
2 for  $m = 1:M$  do   
3 Sort messages based on their  $\gamma$  in a descending order  $S^{\prime} = (S_{1},S_{2},\dots ,S_{N})$  .   
4 for  $l = m - 1:N - 1$  do   
5  $W(m - 1,l) = \max_{l + 1\leq i\leq N}v_iR_i;G(m - 1,l) = \operatorname{argmax}_{l + 1\leq i\leq N}v_iR_i;$    
6 end   
7 for  $k = m - 1:1$  do   
8 for  $l = k - 1:N - m + k - 1$  do   
9  $W(k - 1,l) = \max_{l + 1\leq i\leq N - m + k - 1}(v_iR_i + (1 - v_i)q(m)W(k,i));$    
10  $G(k - 1,l) = \mathrm{argmax}_{l + 1\leq i\leq N - m + k - 1}(v_iR_i + (1 - v_i)q(m)W(k,i))\cup$    
11  $G(k,\mathrm{argmax}_{l + 1\leq i\leq N - m + k - 1}(v_iR_i + (1 - v_i)q(m)W(k,i)))$    
12 end   
13 Output  $W(0,0),G(0,0)$    
14 end   
15 Select the highest  $W(0,0)$  and record the corresponding  $m$  and  $G(0,0)$
Algorithm 1 describes the procedure of finding the optimal list  $\mathbf{S}^*$  with frequency  $f(m^{*})$ . Messages will be first sorted in a descending order based on  $\gamma$ . For any feasible  $m$ , we selected the messages
backwards. The last message must be the one left with the highest expected return  $vR$ . We define
$$
W (k, l) = \sum_ {i = k + 1} ^ {m} v _ {\kappa (i)} R _ {\kappa (i)} \prod_ {s = k + 1} ^ {i - 1} (1 - v _ {\kappa (s)}) q (m), \quad k \leq l \tag {2.1}
$$
as the maximum weighted sum of the last  $(m - k)$  messages' return, when the  $(k + 1)$ -th message to be sent is restricted to be after the  $l$ -th message in  $S'$ .  $G(k, l)$  is to record the messages selected. The transition equation shows the influence of the  $(k - 1)$ -th message on the feasible place for the  $k$ -th messages in  $S'$ . Therefore,  $G(0, 0)$  is the optimal  $m$  messages, and  $W(0, 0)$  means total return. The optimal list is the list with the highest expected return  $W(0, 0)$  among the  $M$  candidates.
3 Learning with Delayed Feedback
In the previous section, we have explored the structure of the optimal list when  $\mathbf{v}$  and  $\mathbf{q}$  are both known. In this section, we turn to the online learning setting with delayed feedback. To differentiate the time stamps, we will use  $r$  to indicate the time when a user arrives (i.e.,  $r = 1,\dots ,T$ ) and use  $t$  to indicate when messages are being sent. The first message for a user  $r$  is sent instantaneously at  $t = r$ , and the subsequent messages are sent at the frequency determined by  $f(m)$ .
A message sent at time  $t$  is delayed if its feedback is received after  $t$ . Let  $\tau$  denote the delayed duration. Throughout this analysis with delayed feedback, we assume that  $0 \leq \tau \leq \tau_{max}$  with probability 1, i.e., the delay has a finite support independent of the attraction probability. When the feedback is not received after  $\tau_{max}$  (e.g., a user stops responding to messages), we assume the user has abandoned. Note that  $\tau_{max}$  is not related to the attraction probability, nor does it imply feedback.
Define  $\mathbf{S}^r$  as the list of messages for user  $r$ , where the total message length is  $m_r$ . We use regret to measure the performance of a learning algorithm, where the regret for a policy  $\pi$  is defined as
$$
R e g (T) = E _ {\pi} \left[ \sum_ {r = 1} ^ {T} U (\mathbf {S} ^ {*}, \mathbf {v}, \mathbf {R}, q (m ^ {*})) - U (\mathbf {S} ^ {r}, \mathbf {v}, \mathbf {R}, q (m _ {r})) \right].
$$
3.1 Learning Algorithm
As the messages are sent sequentially with some time apart, the learning agent may use additional feedback to update the remaining messages. Thus, the content for a user can be dynamic. We will now present our UCB-based learning strategy to achieve this.
We first define our unbiased estimator for  $\mathbf{u}$  and  $\mathbf{q}$  respectively. Let  $T_{i}(t)$  be the total number of feedback (i.e., sum of clicks and no-clicks) received for message  $i$  by time  $t$ , and let  $c_{i}(t)$  denote the number of clicks for message  $i$  by time  $t$ . Similarly, let  $\tilde{T}_{m}(t)$  denote the total number of no-clicks from users with dissemination frequency  $f(m)$ , and  $b_{m}(t)$  denote the number of abandoned users with frequency  $f(m)$  by time  $t$ . Let  $n_{m}(t) = \tilde{T}_{m}(t) - b_{m}(t)$ .
Define the estimator  $\hat{v}_{i,t} = c_i(t) / T_i(t)$  and  $\hat{q}_t(m) = n_m(t) / \tilde{T}_m(t)$ , which are both unbiased. Next, define the upper confidence bound for  $\mathbf{v}$  and  $\mathbf{q}$  as follows,
$$
v _ {i, t} ^ {U C B} = \hat {v} _ {i, t} + \sqrt {2 \frac {\log t}{T _ {i} (t)}}, \text {a n d} q _ {t} ^ {U C B} (m) = \hat {q} _ {t} (m) + \sqrt {2 \frac {\log t}{\tilde {T} _ {m} (t)}}. \tag {3.1}
$$
The following lemma proves the concentration bound for these estimators.
Lemma 3.1. For any  $t$ , we have  $P\left(v_{i,t}^{UCB} - \sqrt{8\frac{\log t}{T_i(t)}} < v_i < v_{i,t}^{UCB}\right) \geq 1 - \frac{2}{t^4}$  for all  $i \in X$  and  $P\left(q_t^{UCB}(m) - \sqrt{8\frac{\log t}{T_m(t)}} < q(m) < q_t^{UCB}(m)\right) \geq 1 - \frac{2}{t^4}$  for all  $1 \leq m \leq M$ .
Then we propose Algorithm 2 for our learning task.
Algorithm 2: An online learning algorithm for cascading bandits with delayed feedback  
1 Initialization: Set  $v_{i,0}^{UCB} = 1$  for all  $i \in X$ ,  $q_0^{UCB}(m) = 1$  for  $m = 1, \dots, M$ ;  $t = 1$ ;  
2 while  $t < T$  do  
3 for Any feedback for some message  $i$  do  
4 Update  $v_{i,t}^{UCB}$  and  $q_t^{UCB}(m)$  according to Equation (3.1);  
5 end  
6 for Any active user  $r$  with message scheduled to be sent at time  $t$  do  
7 Compute  $S^r = \arg \max_{\mathbf{S}} E[U(\mathbf{S}, \mathbf{v}_{t-1}^{UCB}, \mathbf{R}, q_{t-1}^{UCB}(m_r))]$  according to Algorithm 1;  
8 Send the first message  $e$  in  $S^r \backslash O^r$  to user  $r$ ;  $O_r = O_r \cup \{e\}$ ;  
9 end  
10 Compute  $(\mathbf{S}^t, m_t) = \arg \max_{\mathbf{S}, m} E[U(\mathbf{S}, \mathbf{v}_{t-1}^{UCB}, \mathbf{R}, q_{t-1}^{UCB}(m))]$  according to Algorithm 1;  
11 Offer  $S_1^t$  to user  $t$  and the dissemination frequency is set to  $f(m_t)$ ;  $O_t = S_1^t$ ;  
12 end
We introduce the following notations. At time  $t$ , we consider a user  $r$  active where  $r < t$ , if she has not clicked on a message or abandoned, and there is a message scheduled to send to her at time  $t$ . Define  $m_r$  as the total number of messages selected for user  $r$ , where the corresponding abandonment probability is  $1 - q(m_r)$  and the frequency is  $f(m_r)$ . In the recommendation system,  $m_r$  is always fixed for a single user and is determined when she enters the system, but it may vary for users entering the system at different times. Define  $O_r$  as the set of messages that have already been sent to user  $r$ .
Algorithm 2 states that at time  $t$ , the learning agent first updates the UCB estimators for  $\mathbf{v}$  and  $\mathbf{q}$  based on the available information. For the user that just arrived at time  $t$ , the agent determines a list  $\mathbf{S}^t$  with frequency  $f(m_t)$ , and sends the first message  $S_1^t$ . Meanwhile, for an active user  $r$  who is scheduled to have the message sent at time  $t$ , the learning agent determines a new recommending list  $\mathbf{S}^r$  with frequency  $f(m_r)$  (determined at time  $r$ ), and then sends the first message in the list  $\mathbf{S}^r \backslash O^r$ .
3.2 Regret Analysis
Before presenting the regret bound for Algorithm 2, we want to highlight the challenges behind the analysis. Firstly, we allow dynamic content for active users who remain in the system. Secondly, the feedback of individual messages may be delayed for a random amount of time and the frequency of the message dissemination across users could be different. User feedback depends on both the choice of content and the dissemination frequency, which further complicates the analysis. To address these challenges, we develop a novel proof technique to analyze the regret.
Let  $\mathbf{S}^*$  denote the optimal strategy and  $\mathbf{S}^r$  denote the sequence for user  $r$  under strategy  $\pi$ . Without loss of generality, assume  $\gamma_1 \geq \gamma_2 \geq \dots \geq \gamma_N$ . Define  $\mathcal{F}_t$  as the filtration regarding all the information before time  $t$ . We say  $t \in \mathcal{E}_r$  if a message is sent to user  $r$  at time  $t$ .
Lemma 3.2 proves that with the higher attraction probabilities and the lower the abandonment probabilities, the expected payoff is higher. Lemma 3.3 estimates the upper bound regret of a single recommendation under identical reward. Theorem 3.4 gives an upper bound of regret when delayed feedback exists.
Lemma 3.2. Assume  $\mathbf{S}^*$  is the optimal list of messages with length  $m^*$ . Under the condition that  $0 \leq \mathbf{v} \leq \mathbf{v}^{UCB}$  and  $0 \leq q(m^*) \leq q^{UCB}(m^*)$ , we have
$$
E \left[ U \left(\mathbf {S} ^ {*}, \mathbf {v} ^ {U C B}, \mathbf {R}, q ^ {U C B} \left(m ^ {*}\right)\right) \right] \geq E \left[ U \left(\mathbf {S} ^ {*}, \mathbf {v}, \mathbf {R}, q \left(m ^ {*}\right)\right) \right].
$$
Lemma 3.3. When all messages have identical reward, for  $t \in \mathcal{E}_r$  and any  $q' \in (0,1)$ , we have