Title: Cascading Bandits: Optimizing Recommendation Frequency in Delayed Feedback Environments
Venue: NeurIPS 2023
Accepted: yes
Keywords: delayed feedback

recommender system

frequency control
Abstract: Delayed feedback is a critical problem in dynamic recommender systems. In practice, the feedback result often depends on the frequency of recommendation. Most existing online learning literature fails to consider optimization of the recommendation frequency, and regards the reward from each successfully recommended message to be equal. In this paper, we consider a novel cascading bandits setting, where individual messages from a selected list are sent to a user periodically. Whenever a user does not like a message, she may abandon the system with a probability positively correlated with the recommendation frequency.  A learning agent needs to learn both the underlying message attraction probabilities and users' abandonment probabilities through the randomly delayed feedback. We first show a dynamic programming solution to finding the optimal message sequence in deterministic scenarios, in which the reward is allowed to vary with different messages. Then we propose a polynomial time UCB-based offline learning algorithm, and discuss its performance by characterizing its regret bound. For the online setting, we propose a learning algorithm which allows adaptive content for a given user. Numerical experiment on AmEx dataset confirms the effectiveness of our algorithms.

Cascading Bandits: Optimizing Recommendation Frequency in Delayed Feedback Environments

Abstract

Delayed feedback is a critical problem in dynamic recommender systems. In practice, the feedback result often depends on the frequency of recommendation. Most existing online learning literature fails to consider optimization of the recommendation frequency, and regards the reward from each successfully recommended message as equal. In this paper, we consider a novel cascading bandits setting, where individual messages from a selected list are sent to a user periodically. Whenever a user does not like a message, she may abandon the system with a probability positively correlated with the recommendation frequency. A learning agent needs to learn both the underlying message attraction probabilities and users' abandonment probabilities through the randomly delayed feedback. We first show a dynamic programming solution to finding the optimal message sequence in deterministic scenarios, in which the reward is allowed to vary with different messages. Then we propose a polynomial time UCB-based offline learning algorithm, and discuss its performance by characterizing its regret bound. For the online setting, we propose a learning algorithm which allows adaptive content for a given user. Numerical experiment on AmEx dataset confirms the effectiveness of our algorithms.

1 Background

1.1 Introduction

Recommending dynamic content based on adaptive learning of user behavior is a critical task to online platforms. In marketing campaign, as only 2% of web traffic converts on the first visit 1 , it is important to re-target potential users with notifications such as ads, emails or app reminders to gain traction and awareness. The content for those notifications should be dynamic and adapted to a user's behavior. Since the feedback may not be instant, a campaign system still needs to provide campaigns to other users while waiting. Meanwhile, the frequency at which the notifications are sent should also be judiciously managed, i.e., sending more messages may increase the odds of a click, but frequent messaging may also alienate and drive away users.

Online learning for recommendation tasks has been extensively studied in the multi-armed bandit framework. In this setting, pulling certain arms in the bandit problem corresponds to selecting marketing messages for a user. In particular, cascading bandit [13, 14, 18, 21, 20, 25] which models

Junyu Cao ∗

sending a sequence of messages to individual users is a popular choice for analyzing sequential recommendations. However, existing settings in this field are often stylized and overlook certain characteristics that are prevalent in real-world scenarios. For instance, a common assumption is immediate user feedback, whereas in practice, it may take hours to months to receive responses. Additionally, bandit models typically disregard the variation in payback for each message, which limits the practical implications of the model for recommendations. Furthermore, the concept of 'frequency' in message dissemination is absent in current cascading bandit literature, where an entire sequence of messages is sent to a user at once. Consequently, the content for a single user becomes non-adaptive, potentially missing out on additional feedback from other users over an extended time span and foregoing potential benefits.

In this work, we propose and analyze a variant of cascading bandit with five notable differences from the existing literature. Firstly, whenever a user does not like a message, she may abandon the system and forgo receipt of future messages. Thus, the order of messages influences the reward in our setting, which is in contrast to the existing work on cascading bandit such as [13, 14, 18, 21, 20, 25]. Secondly, instead of having the total number of messages in a sequence as a fixed parameter, this quantity is also a decision variable. In particular, we assume the abandonment probability may increase with the frequency at which messages are being disseminated. Thirdly, individual messages for each user are sent out with some time apart, as opposed to all at once. As we gather more feedback from some of the previous messages, we have the ability to update the remaining sequence, and thus the content for a user is dynamic and adaptive. Fourthly, the reward generated after a message is clicked varies with the type of it. Lastly, we also incorporate delayed feedback in our setting.

The analysis for this setting is particularly challenging. One source of complications comes from the interlaced timing of multiple events, i.e., users arrive at different time, messages for individual users are sent at different time, and their responses time is also random. Meanwhile, users' feedback also depends on multiple factors, including the choices of the content, their order, as well as the dissemination frequency.

The contribution of our work is fourfold. 1. We introduce a novel cascading bandit setting with delayed feedback, different reward and frequency control. 2. For the offline combinatorial problem with all known parameters, we characterize a polynomial-time algorithm which determines an optimal list with the appropriate dissemination frequency. 3. For the online setting where we need to learn both the message attraction probabilities and the abandonment probabilities, we propose a learning algorithm and characterized its regret bound. 4. We incorporate personalization by analyzing a contextual variant of the online problem which utilizes both content and user features.

The paper is organized as follows. We review the related literature in Section 1.2. In Section 2, we formally introduce the model and analyze the offline optimization problem with all known parameters. In Section 3 and Section 4, we study the online learning problem in the presence of delayed feedback for the non-contextual and contextual settings, respectively. We evaluate the performance of our algorithms with numerical experiments in Section 5. Lastly, we discuss some limitations and research prospect of our work in Section 6.

1.2 Related Work

The analysis on delayed feedback has started to receive increasing attention in recent years [1, 22, 16, 23]. In online learning, [8] considers a fixed and known delay effect, and shows an additive penalty of the regret for the stochastic setting. [11] analyzes a more general setting and show that the regret increase is multiplicative in the adversarial setting and additive in the stochastic setting. They propose meta-algorithms that can adapt non-delay algorithms and develop a UCB algorithm for the bandit problem. [7] and [19] analyze the impact of delayed feedback in a Gaussian process bandit and an online Markov decision process respectively. Compared to the existing bandit literature, to the best of our knowledge, our work is the first to incorporate delay effects for the cascade model which involves a sequence of actions. Besides the uncertainty contributed by the random response time, the feedback also depends on both the choice of the messages and the order in which they are being sent.

Our work shares some similarity with combinatorial bandits [3, 5, 15, 4] as the learning agent needs to choose a set of messages. However, in our setting, the order of the messages also impacts the rewards. As highlighted in the introduction, our work extends the cascading bandits literature by including frequency control and delayed feedback. This setting can be potentially modeled as a reinforcement

learning problem, such as [24], which proposes an offline reinforcement learning framework to optimize sequential notification decisions. However, this approach significantly increases the learning complexity, making it difficult to analyze performance theoretically.

In the original cascading model [6], a user who is presented with a list of messages examines the content sequentially, until she clicks on the first attractive message. A common different assumption is the dependent click model . In this model, a user may click on multiple messages and finally stop at the truly attractive one. It can be used in bandit settings [12] or generic click-models algorithms [9]. In cascading bandit [13, 14, 18, 21, 20, 25], the attraction probabilities of messages are not known a priori. A learning agent selects a list with a fixed number of messages to incoming users, with the goal to maximize the total clicks over T time steps. User's feedback to a list is immediate. The agent then uses this information to learn the underlying attraction probabilities and updates their list for the next user. Although [10] provides a bound based on Thompson sampling for semi-bandit rewards and [20] provides a problem-independent bound for unstructured rewards, they do not consider the messages' difference in theory. While [2] also considers user abandonment behavior in a cascading bandit setting, the abandonment probability is a constant and the recommendation to a single user is static, as opposed an adaptive sequence which is considered in this work.

2 Problem Formulation

Assume there are N available messages, which we denote as X . We use S = ( S 1 , S 2 , · · · , S m ) to denote a list of messages, where S 1 and S m represent the first and the last message, respectively. Let κ be the index function mapping the position in a list to a message, i.e., κ ( i ) = j if and only if S i = { j } . Define ϑ as the inverse function of κ , i.e., ϑ ( i ) = j if and only if S j = { i } .

Users will receive a list of messages periodically. Upon receiving a message, if a user finds it attractive, she clicks on it. Otherwise, she may examine the next message with probability q ( m ) if the list has not run out, or she may leave with probability 1 -q ( m ) . The probability q ( m ) depends on the total length of the list or the message dissemination frequency, which we will elaborate next. For each message i , we use v i to denote its attraction probability which captures the relevance of its content to a user, and use R i to denote its return when a user clicks on it. We define v = ( v 1 , · · · , v N ) and R = ( R 1 , · · · , R N ) .

A user arrives at time t , where t = 1 , · · · , T . We assume that for every user, there is a fixed time window D , during which selected messages will be disseminated 2 . Messages are sent sequentially at a fixed time interval with a dissemination frequency f ( m ) := ⌈ m/D ⌉ . Here we keep the recommendation frequency fixed for every user, which allows us to easily capture the trade-off between the total number of messages that can be disseminated and the probability of abandonment. At every time step, a user receives at most one message. Define M as the maximum number of messages that can be sent during D . We assume q ( m ) is a non-increasing function with m . The intuition is that when the frequency of re-targeting is higher, a user is more likely to be overwhelmed and is more prone to abandonment. We denote q = ( q (1) , · · · , q ( M )) . The following example illustrates the interactions between users and messages at different time.

Example 2.1. Figure 1 shows 5 users that arrive at time t = 1 , · · · , 5 . Actions taken by users are color-coded, i.e., a user clicks on a specific message (orange), has not clicked but remains active (green), and has abandoned the system (yellow). M i , t = j denotes that message i is sent at time step j , e.g., User 2 receives message 10 at t = 2 and message 9 at t = 6 . While the dissemination frequency is determined for a given user upon arrival, it may be different across users, e.g., User 2 and 5 receive messages at a fixed interval of 4 and 6 time steps respectively.

For a list with m messages, besides the first message which is guaranteed to be viewed, the probability that message i is being examined, denoted as w i ( m ) , depends on the outcomes of the earlier messages, i.e., w i ( m ) = ∏ ϑ ( i ) -1 j =1 ((1 -v κ ( j ) ) q ( m )) . It is the probability that the user does not click on any messages shown before i and still remains in the system. Let U ( S , v , q ( m )) denote the payoff that the learning agent receives from a list of messages S , when the message attraction probabilities are v

t=1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

User 1

M1

M20

M5

User 2

M10

M9

M18

M16

User 3

M15

M21

User 4

M12

M5

M21

M11

and abandonment probability is 1 -q ( m ) . The expected payoff is

To determine a list of messages with appropriate dissemination frequency, the learning agent needs to solve an optimization problem that maximizes E [ U ( S , v , R , q ( m ))] , subject to a constraint that S i ∩ S j = ∅ for i = j , indicating no duplicated messages in the list. We use S ∗ to denote the optimal solution to the combinatorial optimization problem.

̸

In the following result, we characterize the property of the optimal list S ∗ . We then propose an algorithm to identify S ∗ in the offline setting, i.e., when the message attraction probabilities v and the probabilities q describing users' abandonment behavior are known.

Theorem 2.2. In the optimal sequence S ∗ , the characteristic parameter of messages

are sorted in a descending order.

Proof outline: We prove the statement by contradiction. If S is not ordered by γ , we generate another sequence S ′ by swapping a pair of adjacent messages in S . We can prove that the expected payoff of S ′ is higher than S , which is a contradiction.

We include complete proof in the supplementary material. Note that while Theorem 2.2 provides a necessary condition for S ∗ , it does not indicate that the messages with the highest γ should be selected. The structure of E [ U ( S , v , R , q ( m ))] shows that each message in the list will only influence the contribution of all the messages after it on the total return. Therefore, we use a dynamic programming approach to determine the list backwards.

Algorithm 1 describes the procedure of finding the optimal list S ∗ with frequency f ( m ∗ ) . Messages will be first sorted in a descending order based on γ . For any feasible m , we selected the messages

backwards. The last message must be the one left with the highest expected return vR . We define

as the maximum weighted sum of the last ( m -k ) messages' return, when the ( k +1) -th message to be sent is restricted to be after the l -th message in S ′ . G ( k, l ) is to record the messages selected. The transition equation shows the influence of the ( k -1) -th message on the feasible place for the k -th messages in S ′ . Therefore, G (0 , 0) is the optimal m messages, and W (0 , 0) means total return. The optimal list is the list with the highest expected return W (0 , 0) among the M candidates.

3 Learning with Delayed Feedback

In the previous section, we have explored the structure of the optimal list when v and q are both known. In this section, we turn to the online learning setting with delayed feedback. To differentiate the time stamps, we will use r to indicate the time when a user arrives (i.e., r = 1 , · · · , T ) and use t to indicate when messages are being sent. The first message for a user r is sent instantaneously at t = r , and the subsequent messages are sent at the frequency determined by f ( m ) .

A message sent at time t is delayed if its feedback is received after t . Let τ denote the delayed duration. Throughout this analysis with delayed feedback, we assume that 0 ≤ τ ≤ τ max with probability 1, i.e., the delay has a finite support independent of the attraction probability. When the feedback is not received after τ max (e.g., a user stops responding to messages), we assume the user has abandoned. Note that τ max is not related to the attraction probability, nor does it imply feedback.

Define S r as the list of messages for user r , where the total message length is m r . We use regret to measure the performance of a learning algorithm, where the regret for a policy π is defined as

3.1 Learning Algorithm

As the messages are sent sequentially with some time apart, the learning agent may use additional feedback to update the remaining messages. Thus, the content for a user can be dynamic. We will now present our UCB-based learning strategy to achieve this.

We first define our unbiased estimator for u and q respectively. Let T i ( t ) be the total number of feedback (i.e., sum of clicks and no-clicks) received for message i by time t , and let c i ( t ) denote the number of clicks for message i by time t . Similarly, let ˜ T m ( t ) denote the total number of no-clicks from users with dissemination frequency f ( m ) , and b m ( t ) denote the number of abandoned users with frequency f ( m ) by time t . Let n m ( t ) = ˜ T m ( t ) -b m ( t ) .

Define the estimator ˆ v i,t = c i ( t ) /T i ( t ) and ˆ q t ( m ) = n m ( t ) / ˜ T m ( t ) , which are both unbiased. Next, define the upper confidence bound for v and q as follows,

The following lemma proves the concentration bound for these estimators.

Then we propose Algorithm 2 for our learning task.

We introduce the following notations. At time t , we consider a user r active where r < t , if she has not clicked on a message or abandoned, and there is a message scheduled to send to her at time t . Define m r as the total number of messages selected for user r , where the corresponding abandonment probability is 1 -q ( m r ) and the frequency is f ( m r ) . In the recommendation system, m r is always fixed for a single user and is determined when she enters the system, but it may vary for users entering the system at different times. Define O r as the set of messages that have already been sent to user r .

Algorithm 2 states that at time t , the learning agent first updates the UCB estimators for v and q based on the available information. For the user that just arrived at time t , the agent determines a list S t with frequency f ( m t ) , and sends the first message S t 1 . Meanwhile, for an active user r who is scheduled to have the message sent at time t , the learning agent determines a new recommending list S r with frequency f ( m r ) (determined at time r ), and then sends the first message in the list S r \ O r .

3.2 Regret Analysis

Before presenting the regret bound for Algorithm 2, we want to highlight the challenges behind the analysis. Firstly, we allow dynamic content for active users who remain in the system. Secondly, the feedback of individual messages may be delayed for a random amount of time and the frequency of the message dissemination across users could be different. User feedback depends on both the choice of content and the dissemination frequency, which further complicates the analysis. To address these challenges, we develop a novel proof technique to analyze the regret.

Let S ∗ denote the optimal strategy and S r denote the sequence for user r under strategy π . Without loss of generality, assume γ 1 ≥ γ 2 ≥ ·· · ≥ γ N . Define F t as the filtration regarding all the information before time t . We say t ∈ E r if a message is sent to user r at time t .

Lemma 3.2 proves that with the higher attraction probabilities and the lower the abandonment probabilities, the expected payoff is higher. Lemma 3.3 estimates the upper bound regret of a single recommendation under identical reward. Theorem 3.4 gives an upper bound of regret when delayed feedback exists.

Lemma 3.2. Assume S ∗ is the optimal list of messages with length m ∗ . Under the condition that 0 ≤ v ≤ v UCB and 0 ≤ q ( m ∗ ) ≤ q UCB ( m ∗ ) , we have

Lemma 3.3. When all messages have identical reward, for t ∈ E r and any q ′ ∈ (0 , 1) , we have

where e r t,k is the k th message sent to user r at time t .

Theorem 3.4 (Performance bound for Algorithm 2) . The expected regret of Algorithm 2 is bounded above by

for some constants C 1 and C 2 .

Proof Outline : First we note that

Let S r 0 denote the initial list for user r when she just arrives, i.e., S r 0 is the optimal sequence given v UCB r -1 and q UCB r -1 . Note that this list may change at a later time when more information becomes available. Define events B i,t = { v UCB i,t -√ 8 log t/T i ( t ) < v i < v UCB i,t } and E m,t = { q UCB t ( m ) -√ 8 log t/ ˆ T m ( t ) < q ( m ) < q UCB t ( m ) } . Define H t = ⋂ i ∈ X B i,t ⋂ 1 ≤ m ≤ M E m,t . Firstly we could estimate the upper bound of E π [ U ( S ∗ , v , R , q ( m ∗ )) -U ( S ∗ , v , R , q ( m r ))] by coupling. We can get

Regarding the second part in Equation (3.2), we now bound the difference between E π [ U ( S ∗ , v , R , q ( m r ))] and E π [ U ( S r , v , R , q ( m r ))] . Note that S r is an updated list that may differ from S r 0 , so we again use coupling to bound the difference. We couple the processes sending S ∗ and S r , which both have the identical dissemination frequency m r .

Summing over all time steps, we have ∑ T r =1 E π [ U ( S ∗ , v , R , q ( m r )) -U ( S r , v , R , q ( m r ))] ≤ C 4 √ log T ∑ N i =1 E π [ ∑ T t =1 z i,t √ 1 T i ( t -1) ] + DE π [ ∑ T t =1 1( J c t ) ] where J t = ⋂ i ∈ X B i,t . With some derivations, for each i ∈ X , we can bound the term

Combining all the results above, we arrive at the desired result.

The complete proofs of Lemma 3.2, Lemma 3.3, and Theorem 3.4 are included in the supplementary material. If the delay effect is absent, we can show that the regret bound in Theorem 3.4 is reduced to C 1 ( N + M 2 ) √ T log T . In other words, we show that the increase in the regret contributed by the delayed feedback is additive and linear in τ max in our setting. In this paper, we mainly focus on the instance-independent regret, while it is possible to derive an instance-dependent result similarly following Theorem 2 and Theorem 4 in [13], combined with the technique that we have used here to deal with multiple interactions and the delayed feedback.

As a special case of our proposed model, we can compare this result with the state-of-the-art results of cascading bandits. [15] has derived lower bound as Ω( √ NMT ) . Moreover, [11] has shown the regret of O ( √ NT log T + NE [ τ ]) . Our setup is more complicated since there are more unknown parameters that need to be learned (probability of abandonment), and the feedback is received in a delayed manner. The regret bound for our proposed algorithm is O (max( N,M 2 ) √ T + Nτ ) . From the comparison to two special cases, we can see that regret is optimal in T as well as the delay duration.

4 Personalized Recommendation

In the previous section, we assume that all users share the identical preferences towards content and the abandonment behavior. To provide personalized recommendations which incorporate content and

user features, we propose a contextual variant of the learning algorithm which is motivated by [17]. Note that in this section, we allow the message set X to vary for different users, denoted as X r .

Define w r,i as the features of message i in X r , and x r as user's features. We assume the attraction probability of message i is a logit function of its content feature w r,i , and user r 's abandonment behavior can be described by a logit function of the user's features x r . Define the link function µ ( y ) = exp( y ) / (1 + exp( y )) . That is,

Set ˆ Y r,j = 1 if user r remains in the system after she does not click on the j th message in a list, while ˆ Y r,j = 0 if she abandons the system after viewing the j th message. Therefore, the maximum likelihood estimator of α m at time t , ˆ α m,t , can be obtained as

Similarly, let Y r,j = 1 if user r clicks on the message j , and Y r,i = 0 otherwise. The maximum likelihood estimator of β at time t can be obtained as

where feedback includes both clicks and no-clicks and κ r is the index function for user r .

Define the covariance matrices at time t for two estimators as M m,t = ∑ t r =1 ∑ m r j =1 x r x ′ r 1(user r examines the j th message but does no click by time t )1( m r = m ) , and V t = ∑ t r =1 ∑ m r j =1 w r,κ ( j ) w ′ r,κ ( j ) 1(user r gives the feedback for the j th message by time t ). We propose the following contextual learning algorithm. In each step, after observing users' feedback, we update the maximum likelihood estimators for α m and β , respectively, followed by their upper confidence bound. We then use these estimators to determine the messages for the active users.

Algorithm 3: An online algorithm for contextual cascading bandits with delayed feedback

It is also important to note that although Algorithm 2 and Algorithm 3 are both based on the assumption that messages cannot be recommended repeatedly, they can be easily generalized. If

O r is not excluded from the candidate messages, the two algorithms can be applied the repeatrecommendation case. Moreover, if we want to constrain the total number of repetitive displays of one message as B , we can create B copies of that message in the candidate set.

5 Numerical Experiment

In this section, we evaluate the performance of both the non-contextual and contextual algorithms based on the features of the real AmEx User Click dataset 3 , which records over 463,000 recommendations of AmEx from July 2 to July 7 in 2017. The combination of "product"and "campaign_id" is regarded as messages in simulation.

Experiment I: Non-contextual setting. There are N = 25 available messages. The attraction probability v is uniformly generated from distribution [0 , 0 . 5] , and the return R uniformly generated from distribution [1 , 3] , The maximum length of message list M is set to be 10 . Based on estimations from the short lists in AmEx dataset, we set q ( m ) = 1 1+exp(0 . 03 m ) . We set the re-targeting window D = 200 in all settings. The response time τ is uniformly generated from [0 , 3] for each user. In addition to our proposed algorithm, we introduce several algorithms as benchmarks: three different ϵ -greedy algorithms and TS-Cascade, which is a Thompson sampling algorithm for cascading bandits with Gaussian update [25].

Result: We observe the results of 100 independent simulations for T = 30 , 000 time steps. The results are shown in Fig 2, where the shaded area represents the 95% confidence region. We find that the optimistic greedy and decaying ϵ -greedy algorithms are difficult to find an optimal list. The average regret at T = 30 , 000 of our UCB algorithm is 550 . 14 , with a 95% confidence interval of (454 . 71 , 645 . 57) , significantly better than the benchmarks.

0

5000

10000

15000

20000

25000

30000

0

500

1000

1500

2000

2500

Regret

Noncontextual Algorithms Performance Experiment

OptimisticGreedy

DecayingEpsilonGreedy

TS-Cascade

EpsilonGreedy

UCB

Time Steps

0

500

1000

1500

2000

2500

3000

Time Steps

0

100

200

300

400

500

600

Regret

Contextual Algorithms Performance Experiment

ExploreThenCommit

DecayingEpsilonGreedy

LinTS-Cascade

EpsilonGreedy

ContextualUCB

Experiment II: Contextual setting. There are N = 25 available messages. The maximum length of message list M is set to be 20. User features are uniformly generated from [0 , 5] × [0 , 5] × [0 , 5] , and message features are uniformly generated from [ -6 , 0] × [0 , 1] × [0 , 2] × [ -5 , 0] . The coefficient related to the abandonment behavior, denoted as α m , is uniformly generated in the range (fourdimensional including the intercept) -1 . 04 × [ -0 . 064 m, 0] × [ -0 . 08 m, 0] × [ -0 . 16 m, 0] for m = 1 , · · · , 20 , where α m, 1 is the intercept. An alternative coefficient ˜ α m is uniformly generated from -1 . 04 × [ -0 . 004 m, 0] × [ -0 . 064 m, 0] × [ -0 . 08 m, 0] for m = 1 , · · · , 20 . The coefficient related to message attraction is β = (0 . 05 , 0 . 2 , 0 . 1 , 0 . 3 , 0 . 4) , where β 1 is the intercept. Users' response time is uniformly distributed on [0,10]. We also implement several benchmark algorithms: two variants of the ϵ -greedy algorithm, the explore-then-commit algorithm, and LinTS-Cascade, which is a linear generalization of TS-Cascade [25].

Result: We observe the results of 50 independent simulations for T = 3 , 000 time steps. The result is shown in Fig 3. The average regret at T = 3 , 000 of Algorithm 3 is 293 . 94 , with a 95% confidence interval of (273 . 85 , 314 . 04) , significantly better than the benchmark algorithms. In the early stage,

the contextual UCB algorithm also has good performance, showing a faster convergence speed than benchmarks.

Experiment III: Comparison of UCB and contextual UCB. To compare the effect of Algorithm 2 and Algorithm 3, we make two settings: (a): N = 25 , M = 20 with user features and message features generated from the same distribution as in Experiment 2, but only generated once and then fixed. All other parameters are the same as in Experiment 2; (b): same as (a) but with N = 100 .

0

500

1000

1500

2000

2500

3000

0

200

400

600

800

1000

1200

1400

1600

Regret

ContextualUCB vs UCB with Fixed Context Feature

N=100, UCB

N=100, ContextualUCB

N=25, UCB

N=25, ContextualUCB

Time

Result: We observe the results of 50 independent simulations for T = 3 , 000 time steps. The result is shown in Fig 4. The average regrets under the 4 settings are 1451 . 95 , 798 . 94 , 441 . 90 and 280 . 73 from high to low, respectively. The contextual bandit UCB always work better than UCB algorithm in the same setting. Furthermore, the advantage of contextual algorithm becomes greater when there are more messages. This is because more messages will improve the estimation accuracy of α and β .

6 Conclusion

In this work, we considered a cascading bandit problem with delayed feedback and flexible rewards. Based on the users' feedback, the learning agent needs a policy to select a list of messages and determine the frequency at which these messages are disseminated. We proposed a novel learning algorithm to effectively learn such a policy, and it is capable of recommending a list that is dynamic to a given user within polynomial time. Experiments proved the effectiveness and applicability of our algorithm compared to benchmark algorithms in a novel cascading bandit problem.

For future work, it would be interesting to analyze the performance of alternative bandit algorithms, such as Thompson-sampling-based algorithms for this setting. Additionally, considering that users' preferences may change over time, another avenue for future research would be to incorporate non-stationarity into user preferences.

Acknowledgments and Disclosure of Funding

Dairui Wang, Yan Zhang, and Wei Qi acknowledge the support from National Natural Science Foundation of China (Grants 72242106, 72188101).

References

Alekh Agarwal and John C Duchi. Distributed delayed stochastic optimization. In Advances in Neural Information Processing Systems , pages 873-881, 2011.

Junyu Cao and Wei Sun. Dynamic learning of sequential choice bandit problem under marketing fatigue. In The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19) , 2019.

Nicolo Cesa-Bianchi and Gábor Lugosi. Combinatorial bandits. Journal of Computer and System Sciences , 78(5):1404-1422, 2012.

Wei Chen, Yajun Wang, Yang Yuan, and Qinshi Wang. Combinatorial multi-armed bandit and its extension to probabilistically triggered arms. The Journal of Machine Learning Research , 17(1):1746-1778, 2016.

Richard Combes, Mohammad Sadegh Talebi Mazraeh Shahi, Alexandre Proutiere, et al. Combinatorial bandits revisited. In Advances in Neural Information Processing Systems , pages 2116-2124, 2015.

Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. An experimental comparison of click position-bias models. In Proceedings of the 2008 international conference on web search and data mining , pages 87-94. ACM, 2008.

Thomas Desautels, Andreas Krause, and Joel W Burdick. Parallelizing exploration-exploitation tradeoffs in gaussian process bandit optimization. The Journal of Machine Learning Research , 15(1):3873-3923, 2014.

Miroslav Dudik, Daniel Hsu, Satyen Kale, Nikos Karampatziakis, John Langford, Lev Reyzin, and Tong Zhang. Efficient optimal learning for contextual bandits. Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI) , 2011.

Camille-Sovanneary Gauthier, Romaric Gaudel, and Elisa Fromont. UniRank: Unimodal bandit algorithms for online ranking. In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 7279-7309. PMLR, 2022.

Alihan Huyuk and Cem Tekin. Analysis of thompson sampling for combinatorial multi-armed bandit with probabilistically triggered arms. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , volume 89, pages 1322-1330. PMLR, 2019.

Pooria Joulani, Andras Gyorgy, and Csaba Szepesvári. Online learning under delayed feedback. In International Conference on Machine Learning , pages 1453-1461, 2013.

Sumeet Katariya, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. DCM bandits: Learning to rank with multiple clicks. In Proceedings of The 33rd International Conference on Machine Learning , pages 1215-1224. PMLR, 2016.

Branislav Kveton, Csaba Szepesvari, Zheng Wen, and Azin Ashkan. Cascading bandits: Learning to rank in the cascade model. In International Conference on Machine Learning , pages 767-776, 2015.

Branislav Kveton, Zheng Wen, Azin Ashkan, and Csaba Szepesvari. Combinatorial cascading bandits. In Advances in Neural Information Processing Systems , pages 1450-1458, 2015.

Branislav Kveton, Zheng Wen, Azin Ashkan, and Csaba Szepesvari. Tight regret bounds for stochastic combinatorial semi-bandits. In AISTATS , 2015.

John Langford, Alexander Smola, and Martin Zinkevich. Slow learners are fast. arXiv preprint arXiv:0911.0491 , 2009.

Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contextual bandits. In Proceedings of the 34th International Conference on Machine LearningVolume 70 , pages 2071-2080. JMLR. org, 2017.

Shuai Li, Baoxiang Wang, Shengyu Zhang, and Wei Chen. Contextual combinatorial cascading bandits. In ICML , volume 16, pages 1245-1253, 2016.

Gergely Neu, Andras Antos, András György, and Csaba Szepesvári. Online markov decision processes under bandit feedback. In Advances in Neural Information Processing Systems , pages 1804-1812, 2010.

Daniel Vial, Sujay Sanghavi, Sanjay Shakkottai, and R. Srikant. Minimax regret for cascading bandits. In Advances in Neural Information Processing Systems , volume 35, pages 29126-29138, 2022.

Siwei Wang and Wei Chen. Thompson sampling for combinatorial semi-bandits. arXiv preprint arXiv:1803.04623 , 2018.

Marcelo J Weinberger and Erik Ordentlich. On delayed prediction of individual sequences. IEEE Transactions on Information Theory , 48(7):1959-1976, 2002.

Jiaqi Yang and De-Chuan Zhan. Generalized delayed feedback model with post-click information in recommender systems. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 26192-26203. Curran Associates, Inc., 2022.

Yiping Yuan, Ajith Muralidharan, Preetam Nandy, Miao Cheng, and Prakruthi Prabhakar. Offline Reinforcement Learning for Mobile Notifications. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 3614-3623, Atlanta GA USA, October 2022. ACM.

Zixin Zhong, Wang Chi Chueng, and Vincent YF Tan. Thompson sampling algorithms for cascading bandits. The Journal of Machine Learning Research , 22(1):9915-9980, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces several modifications to the cascading bandit setting to make it more suitable for settings where users can provide delayed feedback and abandon the system in response to receiving messages too frequently, such as online ad retargeting. It then provides an algorithm for solving the message queueing optimization problem given a fixed set of parameters describing the user/environment, then a UCB-style algorithm for learning the user/environment parameters while making decisions. It then provides regret bounds for its algorithm and provides experiments for the non-contextual and contextual setting.\n\nSTRENGTHS:\nOriginality/Significance: The authors propose a modification to cascading bandits which allows for delayed feedback and user abandonment, which make it substantially more applicable to several problems such as re-targeting in online advertising.\n\nQuality: The paper includes algorithms for the contextual and non-contextual cases, as well as a regret proof and experiments. Their proposed method is clearly supported by the evidence provided.\n\nClarity: Writing was generally clear.\n\nWEAKNESSES:\n1) The UCB heuristic seems to me like it may be mis-implemented in Algorithms 2 and 3, since it uses the UCB of q instead of the LCB. See question 1 for more details.\n2) The experiments don't compare to a normal cascading bandits baseline.\n3) If I understand correctly, each message can only be used once. While the setting is substantially more realistic than cascading bandits, this still seems like a restrictive assumption.\n\nQUESTIONS:\n1) Should Algorithms 2 and 3 call Algorithm 1 with q^LCB instead of q^UCB? The optimism in the face of uncertainty heuristic is based on always using upper bounds on how good something is, and it seems that the upper bound on $\\gamma$ would be obtained using the lower bound on q instead of the upper bound. For example, in [1] the PrimalDualBwK Algorithm on page 23 uses the lower bound on resource consumption instead of the upper bound.\n2) How important is the restriction to a fixed message frequency for a given message queue length? It seems that most things will email me more at first and then less later.\n3) Is there also an assumption that the user delay is uncorrelated with the click probability? If so, it would be good to mention it since it seems plausible that users who ultimately click are likely to provide feedback sooner.\n\nLIMITATIONS:\nThe limitation of potentially shifting user preferences was mentioned, though I would also re-iterate in the limitations section the uniform dissemination frequency mentioned earlier on in the paper.\n\nThis research seems to have similar societal impact to most other work on bandits applied to online advertising.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studied a generalization setting of the cascading bandit model which has been explored in existing literature. In detail, beyond the existing cascade model, it allows delayed feedback from the user, different number of pulled arms at different time steps, and takes the action frequency into consideration.\nThis work provides and analyzed an algorithm for the setting with delayed feedback and proposed another one for the setting with personalized recommendation.\nNumerical experiments are presented to compare the performance of algorithms.\n\n\n\n====================\n\nI would like to keep the previous score after reading comments from other reviewers and the responses from authors.\n\nSTRENGTHS:\n1. This paper formulates a more realistic cascading bandit model. In detail, it generalized the existing model by taking delayed feedback, flexible length of recommendation list and action frequency into consideration.\n3. The motivation to study this novel setting is well explained and convincing.\n2. To help readers understand the formulation of the problem, Example 2.1 with Figure 1 provides a vivid instance.\n\nWEAKNESSES:\n1. The regret upper bound for Algorithm 2 is provided in Theorem 3.3, I think a detailed comparison to existing bounds should be provided here: \n    1. I guess Algorithm 2 also works for the traditional cascading bandit setting, how will the regret bound be in the traditional setting? Does it match the state-of-the-art bounds?\n    2. What is the corresponding lower bound in this novel setting? Are the upper/lower bounds tight?\n2. What is the key challenge to derive a bound for Algorithm 3 in the setting with personalized recommendation?\n3. In numerical experiments, Thompson sampling algorithms are not mentioned. I believe the authors should also compare Thompson sampling algorithms or explain the reason not to do so.\n4. The literature review does not seem to be complete. I think the author(s) may consider to mention the following works on cascading bandits:\n    1. Minimax Regret for Cascading Bandits https://proceedings.neurips.cc/paper_files/paper/2022/hash/bbb397739546de027bc81a2c2a8fb119-Abstract-Conference.html\n    1. Thompson Sampling Algorithms for Cascading Bandits\nhttps://jmlr.csail.mit.edu/papers/volume22/20-447/20-447.pdf\n    1. Analysis of Thompson sampling for combinatorial multiarmed bandit with probabilistically triggered arms\nhttps://proceedings.mlr.press/v89/huyuk19a.html\n\nQUESTIONS:\nMajor concerns are listed in the *Weakness* section, and some possible typos are as below:\n1. Line 26: 'multi-arm bandit' should be 'multi-armed bandit'\n2. Line 81: 'our work extends the cascading bandits literature include' should be 'our work extends the cascading bandits literature by including' \n3. Line 85: 'known a priori' should be 'known as a priori'\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a new variant of cascading bandit that differs with the classical cascading bandit in the following ways: (1). the user might abandon the system with certain probability depending on the list length; (2). the list length is some variable to be optimized (even could be contextualized), instead of fixed in advance; and (3). the list could be adaptively updated as more user feedbacks coming in. This new variation is closer to real-world settings such as notification/personalized recommendation systems. \n\nThe paper proposes a dynamic programming approach for finding the optimal length and optimal list periodically and extend it to the contextualized setting where the relevance and abandon probability is a function of contextual features. Some simple synthetic experiments verify the effectiveness of the method compared with epsilon-greedy based approach.\n\nSTRENGTHS:\n- The setting this paper studies, extending from classical cascading bandit, is closer to real-world applications. The modeling/assumptions are realistic and not deviate that much more from the use-cases. I do like the new formulation. \n\n- The paper provides theoretical understanding of the non-contextualized case, the dynamic programming approach is intuitive and interesting, the regret compared with the optimal list and list length is given. The paper further extends it to the non-contextualized case, allowing parametric form of the relevance and abandon probability. The method is solid and the analysis seems valid. \n\n- The paper is very well-written. It motivates the problem in a crystal clear way and thoroughly listed the difference compared with the previous approach. I enjoy reading the paper.\n\nWEAKNESSES:\n- One thing i am concerned about is the computational feasibility of the method. It seems for each user, the exact DP is performed, which might be super expensive when the length list $m$ is large. Could the authors comment more on the computational aspect?\n\n- For the experiments, the author mentioned that the features is based on the real-world dataset, but the model parameters are being simulated. I am wondering how the real-world features help here? It seems to me closer to a fully synthetic set-up. Could the authors comment more on this aspect?\n\n- Following this, it seems the experiment is performed in a very small scale. In real-world scenarios, the number of available messages might be very large, and also the length of the list. Could the authors compare the proposed method in a larger scale experiments, preferably more real-world scenarios?\n\n- It seems there are also RL based approach for solving the notification problem [1], have the authors thought of this aspect and it would be great to add literature review on this aspect and discuss the pros/cons, even empirical comparison? \n\nRef: [1]: https://arxiv.org/abs/2202.03867\n\nQUESTIONS:\n- Do we need to have a fixed list of messages? How the algorithms adapted to the new possible messages that arrive at different time-steps?\n\n- One thing confused me is that whether the learned list length $m$ is updated in each DP? or it is only computed and fixed on the first round?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a bandit setting where the agent selects a list of messages to be sent one at a time in a time window of length $D$. For each message, the user may be attracted (and sent a reward) or not (in which case she may leave the system with a probability that depends on the frequency of the messages).\n\nThe authors propose an algorithm to identify the optimal sequence of messages (including their size) if the parameters are known, and use it to derive two UCB-like algorithms dedicated to this setting. The first algorithm treats the messages and users as independent entities, while the second is dedicated to the contextual setting, where the attraction probabilities depend on known characteristics of the messages and the abandonment probability depends on known characteristics of the user.\n\nThe regret of the first algorithm is theoretically bounded, and some experiments compare both algorithms and epsilon-greedy.\n\nSTRENGTHS:\nThe main strengths of the paper are the proposed setting and the derivation of the optimal solution for this setting when the parameters are known.  \nRegarding the setting, the notion of delayed feedback is not new. However, (i) the sending of messages one at a time, (ii) the dependence of the abandonment probability on the message frequency, and (iii) the distinction between attraction and reward, are interesting from an application point of view.\n\nTwo UCB-like algorithms are proposed, but they are simple.\n\nSimilarly, the experiences are interesting but remain of little impact since the setting is new and there is no baseline apart from $\\epsilon$-greedy.\n\nWEAKNESSES:\nThe theoretical analysis is limited to the worst-case regret and covers only the algorithm for independent arms.\n\nApart from this weakness,  I have a comment: The literature on click models includes a click model with probability of leaving the system: the *dependent click\nmodel*. This model is used in the bandit setting in \n\n> S. Katariya, B. Kveton, Cs. Szepesvári, and Z. Wen. DCM bandits: Learning to rank with multiple clicks. In Proceedings of the 33rd International Conference on Machine Learning, pages 1215–1224, 2016.\n\nor by generic click-models algorithms such as TopRank (cited in the appendix) or UniRank\n\n> Camille-Sovanneary GAUTHIER · Romaric Gaudel · Elisa Fromont. UniRank: Unimodal Bandit Algorithms for Online Ranking. ICML'22\n\nQUESTIONS:\nDo you know how to derive an instance-dependent proof  (and a proof of the corresponding lower-bound) ?\n\nLIMITATIONS:\nThere is no section dedicated to the limits of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}