Bandit Learning with Joint Effect of Incentivized Sampling, Delayed Sampling Feedback, and Self-Reinforcing User Preferences

1 INTRODUCTION . In recent years , the multi-armed bandit ( MAB ) framework has received a significant amount of interest in the learning research community . This is partly due to the fact that , in many online ecommerce recommender systems ( e.g. , Amazon and Walmart ) , the problem of online learning of the optimal products while making profits at the same time can be well formulated by an MAB problem . However , although many MAB algorithms have been proposed in this area , it is worth noting that most of the existing MAB models in the literature have not considered the joint effect of three common phenomena in e-commerce recommender systems : ( i ) In many e-commerce recommender systems , the platform ( the learning agent ) can not sample an intended product ( an intended arm ) directly and has to incentivize customers ( e.g. , through promotions and coupons ) to sample the product and receive the sampling feedback from the customers indirectly ( e.g. , ratings and reviews ) ; ( ii ) Customer feedbacks are often received much later than their purchasing times ( e.g. , a review may or may not be submitted by a customer even months later after purchasing a product ) ; and ( iii ) Customer preferences among products are influenced and reinforced by historical feedbacks , which may even lead to various viral effects over some products ( the more good reviews one product has received , the more likely that the next arriving customer will prefer this product ) . The lack of a fundamental understanding and joint studies of these three important factors in MAB policy designs motivates us to fill this gap in this paper . Toward this end , we propose a new MAB framework that jointly considers i ) incentivized sampling , ii ) delayed sampling feedback , and iii ) self-reinforcing user preferences in online recommender systems . However , we note that the MAB policy design for the proposed new MAB framework is highly non-trivial due to the complex couplings between the aforementioned three factors . First , similar to conventional MAB problems , there exists a dilemma between sufficient exploration through sampling to learn an optimal arm ( i.e. , an optimal product ) , which may incur numerous pullings of sub-optimal arms , and the greedy exploitation to play the arm that has performed well thus far to earn profits . Second , there is another dilemma to the learning agent between offering sufficiently attractive incentives to mitigate biases ( due to lack of initial data and self-reinforcing user preferences ) and avoid spending unnecessarily high incentives that hurt the learning agent ’ s profits . Last but not least , the delayed sampling feedbacks may render the estimation of arms ’ quality during the MAB process highly inaccurate , introducing yet another layer of uncertainty to the MAB online learning problem , which is already plagued by complications from incentivized sampling and self-reinforcing user preferences . As in most MAB problems , we adopt “ regret ” as our performance metric in this paper , which is defined as the cumulative reward gap between the proposed policy and an optimal policy design in hindsight . Under the regret setting , the complications due to these three key factors naturally prompt the following fundamental questions : ( 1 ) How should the agent design an incentivizing strategy to strike a good balance between explo- ration and exploitation to achieve sublinear ( hopefully logarithmic ) regrets ? ( 2 ) To avoid offering exceedingly high incentives , how should the agent incentivize in order to attract a user crowd that prefer an optimal arm , so that the users ’ self-reinforcing preference could automatically gravitate toward this optimal arm without further incentives ? ( 3 ) Under various delayed feedback situations in the new MAB framework ( e.g. , unbounded random delays , heavy-tailed delay distributions , and arm-dependent delays ) , could we still achieve low regrets with low incentive costs ? In this paper , we answer the above fundamental questions affirmatively by proposing a new “ Delayed-UCB-Filtering ” policy for the MAB framework that jointly considers incentivizing sampling , delayed sampling feedback , and self-reinforcing user preferences . We show that our proposed policy achievesO ( log T ) regret withO ( log T ) incentive payments . The success of our policy design hinges upon two key insights : ( i ) the self-reinforcing user preference effect is actually a “ blessing in disguise ” and can be leveraged to establish an important “ dominance ” condition ( more on this later ) that further implies O ( log T ) regret and incentive costs ; and ( ii ) the impacts of delayed feedback on regret and incentive costs can be upper bounded under appropriate statistical settings to preserve the “ dominance ” condition . Our key contributions and main results are summarized as follows : • We propose a new MAB model that jointly considers incentivized arm sampling , delayed sam- pling feedback , and self-reinforcing user preferences , all of which are important features of online recommender systems . To develop efficient and low-cost incentivized policy for this new MAB model , we propose a three-phase “ UCB-Filtering-with-Delayed-Feedback ” ( UCB-FDF ) policy , which contains an incentivized exploration phase , an incentivized exploitation phase , and a selfsustaining phase . In our UCB-FDF policy , the first two phases judiciously integrate delayed feedback information , while in third phase , the system solely relies on self-reinforcing user preferences to converge to the pulling of the optimal arm . • We first show a fundamental fact that , under our UCB-FDF policy , delayed sampling feedback only has an additive penalty on the regret and incentive cost performances , and that this additive penalty grows logarithmically with respect to time . Specifically , we first investigate the delayed feedback impact under the assumption that the feedback delay is an i.i.d . random variable across samplings with a finite expectation . We show that the UCB-FDF policy achieves logarithmic growth rates of regret and incentive costs under this setting . Then , we relax the i.i.d . feedback delay assumption to allow the feedback delay distribution to be arm-dependent . Under this setting , we also show that similar logarithmic growth rates of regret and incentive can still be achieved . • We conduct extensive experiments on Amazon Review Data 1 to demonstrate and verify the performance of our UCB-FDF policy as well as the impacts of delayed feedback on real-world scenarios . We also verify our theoretical analysis through various product categories and demonstrate the efficacy of our proposed UCB-FDF MAB policy . The rest of the paper is organized as follows . In Section 2 , we review the literature to put our work in comparative perspectives . Then in Section 3 , we formulate our new MAB model that capture the three common phenomena . In Section 4 , we present our UCB-FDF policy and analyze its performance . Then , we present our experiment settings and results in Section 5 . Due to space limitations , the proofs are relegated to the appendix . 1https : //nijianmo.github.io/amazon/ 2 RELATED WORK . In this section , we provide a quick overview on three lines of research related to our work : i ) bandits with delayed feedback , ii ) bandits with random preferences , and iii ) incentivized bandits . 1 ) Bandits with Delayed Feedback : Motivated by practical issues in the clinical trials , Eick ( 1988 ) was the first to introduce a two-armed bandit model with delayed responses , where the patients survival time reports after the treatment are delayed . Recently , Joulani et al . ( 2013 ) provided a systematic study and showed that for delay τ with a finite expectation , the worst case regret scales with O ( √ KT log T+KE [ τ ] ) , whereK is the number of arms . Meanwhile , Vernade et al . ( 2017 ) showed that stochastic MAB problems with delayed feedback have a regret lower bound O ( K log T ) . However , this work assumed that the distribution of the random delay is arm-independent . In contrast , Joulani et al . ( 2013 ) considered arm-dependent delay distributions that have an upper bound of the maximum random delay . More recently , Manegueu et al . ( 2020 ) considered arm-dependent and heavy-tailed delay distributions , where only an upper bound on the tail of the delay distribution is needed , without requiring the expectation to be finite . Also , Lancewicki et al . ( 2021 ) studied the case where the delay distribution is reward-dependent , which implies that the random delay in each round may also depend on the reward received on the same round . However , most of these works on delayed bandits are based on the standard stochastic MAB framework . In contrast , we consider delayed feedback in incentivized bandit learning with self-reinforcing user preferences , which is a more appropriate model for real-world recommender systems than the standard stochastic MAB . 2 ) Bandits with Random Preferences : The impacts of random user preferences in e-commerce platforms have received increasing interest in several different areas in learning and economics . Existing works in ( Agrawal et al. , 2017 ; 2019 ) formulated the user preference variation given different product bundles by the multi-nomial logit model on top of the bandit learning framework and proposed a Thompson Sampling approach that achieves a worst-case regret bound ofO ( √ NT log TK ) , where N is the size of recommended arm bundle . With a different focus on preference modeling , Barabási & Albert ( 1999 ) ; Chakrabarti et al . ( 2006 ) ; Ratkiewicz et al . ( 2010 ) investigated the network evolution with “ preferential attachment ” that formulates the social behavior known as selfreinforcing preferences , among which the works in ( Shah et al. , 2018 ; Zhou et al. , 2021 ) are the closet to our work . To our knowledge , Shah et al . ( 2018 ) was the first to consider self-reinforcing user preferences in bandit learning problems . Later , Zhou et al . ( 2021 ) incorporated self-reinforcing user preferences into the incentivized bandit learning framework . The key difference between these two works is that , in the model in ( Shah et al. , 2018 ) , only one arm is revealed to users in each round , while in the model in ( Zhou et al. , 2021 ) , all arms are revealed to users and users ’ arm selections are influenced by incentives . However , both of these works fall short in modeling online recommender systems in practice as they assume that an arm-sampling feedback is observable in the same timeslot when an arm is pulled . However , for most e-commerce recommender systems in practice , user feedbacks are often not immediately observable . As a result , the decision on which arm to pull next has to be made without some of the feedbacks from arm-pulling actions in the past . 3 ) Incentivized Bandits : To our knowledge , Frazier et al . ( 2014 ) was among the first to adopt incentive schemes into a Bayesian MAB setting . In their model , the agent seeks to maximize timediscounted total reward by incentivizing arm selections . Later , Mansour et al . ( 2015 ) studied the non-discounted reward setting . For the non-Bayesian setting , Wang & Huang ( 2018 ) ; Zhou et al . ( 2021 ) proposed policies that maximize the total non-discounted reward with bounded incentive costs . Bandits with budget ( Guha & Munagala , 2007 ; Goel et al. , 2009 ) also share some similarities with our work , where the agent takes actions under resource constraints that are either fixed or with a given growth rate bound . However , none of the aforementioned works considered the impacts of delayed feedback on the regret and incentive costs performances . Note that , due to the loss of information caused by delayed feedback , larger variances in the mean-reward estimations of the arms are inevitable . This implies that , in order to achieve a more accurate arm quality estimation under delayed feedbacks , a higher incentive cost is necessary .

Summary: This paper studies a stochastic multi-armed bandit setting with delayed feedback, where additionally, the bandit policy must incentivize an external agent to pull the desired arm, and the arm preferences of this external agent are self-reinforcing. They design a policy, UCB-FDF, for this setting, and prove expected regret (and incentive payment) upper bounds under various delay distribution assumptions. Finally, they evaluate their policy on bandit environments constructed using Amazon review data.