Bandit Learning with Joint Effect of Incentivized Sampling, Delayed Sampling Feedback, and Self-Reinforcing User Preferences

1 INTRODUCTION . In recent years , the multi-armed bandit ( MAB ) framework has received a significant amount of interest in the learning research community . This is partly due to the fact that , in many online ecommerce recommender systems ( e.g. , Amazon and Walmart ) , the problem of online learning of the optimal products while making profits at the same time can be well formulated by an MAB problem . However , although many MAB algorithms have been proposed in this area , it is worth noting that most of the existing MAB models in the literature have not considered the joint effect of three common phenomena in e-commerce recommender systems : ( i ) In many e-commerce recommender systems , the platform ( the learning agent ) can not sample an intended product ( an intended arm ) directly and has to incentivize customers ( e.g. , through promotions and coupons ) to sample the product and receive the sampling feedback from the customers indirectly ( e.g. , ratings and reviews ) ; ( ii ) Customer feedbacks are often received much later than their purchasing times ( e.g. , a review may or may not be submitted by a customer even months later after purchasing a product ) ; and ( iii ) Customer preferences among products are influenced and reinforced by historical feedbacks , which may even lead to various viral effects over some products ( the more good reviews one product has received , the more likely that the next arriving customer will prefer this product ) . The lack of a fundamental understanding and joint studies of these three important factors in MAB policy designs motivates us to fill this gap in this paper . Toward this end , we propose a new MAB framework that jointly considers i ) incentivized sampling , ii ) delayed sampling feedback , and iii ) self-reinforcing user preferences in online recommender systems . However , we note that the MAB policy design for the proposed new MAB framework is highly non-trivial due to the complex couplings between the aforementioned three factors . First , similar to conventional MAB problems , there exists a dilemma between sufficient exploration through sampling to learn an optimal arm ( i.e. , an optimal product ) , which may incur numerous pullings of sub-optimal arms , and the greedy exploitation to play the arm that has performed well thus far to earn profits . Second , there is another dilemma to the learning agent between offering sufficiently attractive incentives to mitigate biases ( due to lack of initial data and self-reinforcing user preferences ) and avoid spending unnecessarily high incentives that hurt the learning agent ’ s profits . Last but not least , the delayed sampling feedbacks may render the estimation of arms ’ quality during the MAB process highly inaccurate , introducing yet another layer of uncertainty to the MAB online learning problem , which is already plagued by complications from incentivized sampling and self-reinforcing user preferences . As in most MAB problems , we adopt “ regret ” as our performance metric in this paper , which is defined as the cumulative reward gap between the proposed policy and an optimal policy design in hindsight . Under the regret setting , the complications due to these three key factors naturally prompt the following fundamental questions : ( 1 ) How should the agent design an incentivizing strategy to strike a good balance between explo- ration and exploitation to achieve sublinear ( hopefully logarithmic ) regrets ? ( 2 ) To avoid offering exceedingly high incentives , how should the agent incentivize in order to attract a user crowd that prefer an optimal arm , so that the users ’ self-reinforcing preference could automatically gravitate toward this optimal arm without further incentives ? ( 3 ) Under various delayed feedback situations in the new MAB framework ( e.g. , unbounded random delays , heavy-tailed delay distributions , and arm-dependent delays ) , could we still achieve low regrets with low incentive costs ? In this paper , we answer the above fundamental questions affirmatively by proposing a new “ Delayed-UCB-Filtering ” policy for the MAB framework that jointly considers incentivizing sampling , delayed sampling feedback , and self-reinforcing user preferences . We show that our proposed policy achievesO ( log T ) regret withO ( log T ) incentive payments . The success of our policy design hinges upon two key insights : ( i ) the self-reinforcing user preference effect is actually a “ blessing in disguise ” and can be leveraged to establish an important “ dominance ” condition ( more on this later ) that further implies O ( log T ) regret and incentive costs ; and ( ii ) the impacts of delayed feedback on regret and incentive costs can be upper bounded under appropriate statistical settings to preserve the “ dominance ” condition . Our key contributions and main results are summarized as follows : • We propose a new MAB model that jointly considers incentivized arm sampling , delayed sam- pling feedback , and self-reinforcing user preferences , all of which are important features of online recommender systems . To develop efficient and low-cost incentivized policy for this new MAB model , we propose a three-phase “ UCB-Filtering-with-Delayed-Feedback ” ( UCB-FDF ) policy , which contains an incentivized exploration phase , an incentivized exploitation phase , and a selfsustaining phase . In our UCB-FDF policy , the first two phases judiciously integrate delayed feedback information , while in third phase , the system solely relies on self-reinforcing user preferences to converge to the pulling of the optimal arm . • We first show a fundamental fact that , under our UCB-FDF policy , delayed sampling feedback only has an additive penalty on the regret and incentive cost performances , and that this additive penalty grows logarithmically with respect to time . Specifically , we first investigate the delayed feedback impact under the assumption that the feedback delay is an i.i.d . random variable across samplings with a finite expectation . We show that the UCB-FDF policy achieves logarithmic growth rates of regret and incentive costs under this setting . Then , we relax the i.i.d . feedback delay assumption to allow the feedback delay distribution to be arm-dependent . Under this setting , we also show that similar logarithmic growth rates of regret and incentive can still be achieved . • We conduct extensive experiments on Amazon Review Data 1 to demonstrate and verify the performance of our UCB-FDF policy as well as the impacts of delayed feedback on real-world scenarios . We also verify our theoretical analysis through various product categories and demonstrate the efficacy of our proposed UCB-FDF MAB policy . The rest of the paper is organized as follows . In Section 2 , we review the literature to put our work in comparative perspectives . Then in Section 3 , we formulate our new MAB model that capture the three common phenomena . In Section 4 , we present our UCB-FDF policy and analyze its performance . Then , we present our experiment settings and results in Section 5 . Due to space limitations , the proofs are relegated to the appendix . 1https : //nijianmo.github.io/amazon/ 2 RELATED WORK . In this section , we provide a quick overview on three lines of research related to our work : i ) bandits with delayed feedback , ii ) bandits with random preferences , and iii ) incentivized bandits . 1 ) Bandits with Delayed Feedback : Motivated by practical issues in the clinical trials , Eick ( 1988 ) was the first to introduce a two-armed bandit model with delayed responses , where the patients survival time reports after the treatment are delayed . Recently , Joulani et al . ( 2013 ) provided a systematic study and showed that for delay τ with a finite expectation , the worst case regret scales with O ( √ KT log T+KE [ τ ] ) , whereK is the number of arms . Meanwhile , Vernade et al . ( 2017 ) showed that stochastic MAB problems with delayed feedback have a regret lower bound O ( K log T ) . However , this work assumed that the distribution of the random delay is arm-independent . In contrast , Joulani et al . ( 2013 ) considered arm-dependent delay distributions that have an upper bound of the maximum random delay . More recently , Manegueu et al . ( 2020 ) considered arm-dependent and heavy-tailed delay distributions , where only an upper bound on the tail of the delay distribution is needed , without requiring the expectation to be finite . Also , Lancewicki et al . ( 2021 ) studied the case where the delay distribution is reward-dependent , which implies that the random delay in each round may also depend on the reward received on the same round . However , most of these works on delayed bandits are based on the standard stochastic MAB framework . In contrast , we consider delayed feedback in incentivized bandit learning with self-reinforcing user preferences , which is a more appropriate model for real-world recommender systems than the standard stochastic MAB . 2 ) Bandits with Random Preferences : The impacts of random user preferences in e-commerce platforms have received increasing interest in several different areas in learning and economics . Existing works in ( Agrawal et al. , 2017 ; 2019 ) formulated the user preference variation given different product bundles by the multi-nomial logit model on top of the bandit learning framework and proposed a Thompson Sampling approach that achieves a worst-case regret bound ofO ( √ NT log TK ) , where N is the size of recommended arm bundle . With a different focus on preference modeling , Barabási & Albert ( 1999 ) ; Chakrabarti et al . ( 2006 ) ; Ratkiewicz et al . ( 2010 ) investigated the network evolution with “ preferential attachment ” that formulates the social behavior known as selfreinforcing preferences , among which the works in ( Shah et al. , 2018 ; Zhou et al. , 2021 ) are the closet to our work . To our knowledge , Shah et al . ( 2018 ) was the first to consider self-reinforcing user preferences in bandit learning problems . Later , Zhou et al . ( 2021 ) incorporated self-reinforcing user preferences into the incentivized bandit learning framework . The key difference between these two works is that , in the model in ( Shah et al. , 2018 ) , only one arm is revealed to users in each round , while in the model in ( Zhou et al. , 2021 ) , all arms are revealed to users and users ’ arm selections are influenced by incentives . However , both of these works fall short in modeling online recommender systems in practice as they assume that an arm-sampling feedback is observable in the same timeslot when an arm is pulled . However , for most e-commerce recommender systems in practice , user feedbacks are often not immediately observable . As a result , the decision on which arm to pull next has to be made without some of the feedbacks from arm-pulling actions in the past . 3 ) Incentivized Bandits : To our knowledge , Frazier et al . ( 2014 ) was among the first to adopt incentive schemes into a Bayesian MAB setting . In their model , the agent seeks to maximize timediscounted total reward by incentivizing arm selections . Later , Mansour et al . ( 2015 ) studied the non-discounted reward setting . For the non-Bayesian setting , Wang & Huang ( 2018 ) ; Zhou et al . ( 2021 ) proposed policies that maximize the total non-discounted reward with bounded incentive costs . Bandits with budget ( Guha & Munagala , 2007 ; Goel et al. , 2009 ) also share some similarities with our work , where the agent takes actions under resource constraints that are either fixed or with a given growth rate bound . However , none of the aforementioned works considered the impacts of delayed feedback on the regret and incentive costs performances . Note that , due to the loss of information caused by delayed feedback , larger variances in the mean-reward estimations of the arms are inevitable . This implies that , in order to achieve a more accurate arm quality estimation under delayed feedbacks , a higher incentive cost is necessary .

Summary: This paper combines three aspects of MAB: delayed reward, incentivized exploration and self-reinforcing user preference. They motivate this problem from the perspective of online recommender systems. For this model, they propose a new UCB based algorithm that achieves the optimal upper bounds. They also setup an online experiment based on Amazon review data and show how the regret evolves for both arm-independent delays and arm-dependent delays.