Rebounding Bandits for Modeling Satiation Effects

1 Introduction . Recommender systems suggest such diverse items as music , news , restaurants , and even job candidates . Practitioners hope that by leveraging historical interactions , they might provide services better aligned with their users ’ preferences . However , despite their ubiquity in application , the dominant learning framework suffers several conceptual gaps that can result in misalignment between machine behavior and human preferences . For example , because human preferences are seldom directly observed , these systems are typically trained on the available observational data ( e.g. , purchases , ratings , or clicks ) with the objective of predicting customer behavior [ 4 , 27 ] . Problematically , such observations tend to be confounded ( reflecting exposure bias due to the current recommender system ) and subject to censoring ( e.g. , users with strong opinions are more likely to write reviews ) [ 41 , 16 ] . Even if we could directly observe the utility experienced by each user , we might expect it to depend , in part , on the history of past items consumed . For example , consider the task of automated ( music ) playlisting . As a user is made to listen to the same song over and over again , we might expect that the utility derived from each consecutive listen would decline [ 35 ] . However , after listening to other music for some time , we might expect the utility associated with that song to bounce back towards its baseline level . Similarly , a diner served pizza for lunch might feel diminished pleasure upon eating pizza again for dinner . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The psychology literature on satiation formalizes the idea that enjoyment depends not only on one ’ s intrinsic preference for a given product but also on the sequence of previous exposures and the time between them [ 3 , 6 ] . Research on satiation dates to the 1960s ( if not earlier ) with early studies addressing brand loyalty [ 42 , 28 ] . Interestingly , even after controlling for marketing variables like price , product design , promotion , etc. , researchers still observe brand-switching behavior in consumers . Such behavior , referred as variety seeking , has often been explained as a consequence of utility associated with the change itself [ 25 , 17 ] . For a comprehensive review on hedonic decline caused by repeated exposure to a stimulus , we refer the readers to [ 11 ] . In this paper , we introduce rebounding bandits , a multi-armed bandits ( MABs ) [ 37 ] framework that models satiation via linear dynamical systems . While traditional MABs draw rewards from fixed but unknown distributions , rebounding bandits allow each arm ’ s rewards to evolve as a function of both the per-arm characteristics ( susceptibility to satiation and speed of rebounding ) and the historical pulls ( e.g. , past recommendations ) . In rebounding bandits , even if the dynamics are known and deterministic , selecting the optimal sequence of T arms to play requires planning in a Markov decision process ( MDP ) whose state space scales exponentially in the horizon T . When the satiation dynamics are known and stochastic , the states are only partially observable , since the satiation of each arm evolves with ( unobserved ) stochastic noises between pulls . And when the satiation dynamics are unknown , learning requires that we identify a stochastic dynamical system . We propose Explore-Estimate-Plan ( EEP ) an algorithm that ( i ) collects data by pulling each arm repeatedly , ( ii ) estimates the dynamics using this dataset ; and ( iii ) plans using the estimated parameters . We provide guarantees for our estimators in § 6.2 and bound EEP ’ s regret in § 6.3 . Our main contributions are : ( i ) the rebounding bandits problem ( §3 ) , ( ii ) analysis showing that when arms share rewards and ( deterministic ) dynamics , the optimal policy pulls arms cyclically , exhibiting variety-seeking behavior ( §4.1 ) ; ( iii ) an estimator ( for learning the satiation dynamics ) along with a sample complexity bound for identifying an affine dynamical system using a single trajectory of data ( §6.2 ) ; ( iv ) EEP , an algorithm for learning with unknown stochastic dynamics that achieves sublinear w-step lookahead regret [ 34 ] ( §6 ) ; and ( v ) experiments demonstrating EEP ’ s efficacy ( §7 ) . 2 Related Work . Satiation effects have been addressed by such diverse disciplines as psychology , marketing , operations research , and recommendation systems . In the psychology and marketing literatures , satiation has been proposed as an explanation for variety-seeking consumer behavior [ 11 , 25 , 26 ] . In operations research , addressing continuous consumption decisions , [ 3 ] propose a deterministic linear dynamical system to model satiation effects . In the recommendation systems community , researchers have used semi-Markov models to explicitly model two states : ( i ) sensitization—where the user is highly interested in the product ; and ( ii ) boredom—where the user is not engaged [ 18 ] . The bandits literature has proposed a variety of extensions where rewards depend on past exposures , both to address satiation and other phenomena . [ 14 , 21 , 39 ] tackle settings where each arm ’ s expected reward grows ( or shrinks ) monotonically in the number of pulls . By contrast , [ 19 , 2 , 7 ] propose models where rewards increase as a function of the time elapsed since the last pull . [ 34 ] model the expected reward as a function of the time since the last pull drawn from a Gaussian Process with known kernel . [ 43 ] propose a model where rewards are linear functions of the recent history of actions and [ 29 ] model the reward as a function of a context that evolves according to known deterministic dynamics . In rested bandits [ 12 ] , an arm ’ s expected rewards change only when it is played , and in restless bandits [ 44 ] rewards evolve independently from the play of each arm . Key Differences This may be the first bandits paper to model evolving rewards through continuousstate linear stochastic dynamical systems with unknown parameters . Our framework captures several important aspects of satiation : rewards decline by diminishing amounts with consecutive pulls and rebound towards the baseline with disuse . Unlike models that depend only on fixed windows or the time since the last pull , our model expresses satiation more organically as a quantity that evolves according to stochastic dynamics . To estimate the reward dynamics , we leverage recent advances in the identification of linear dynamical systems [ 40 , 38 ] that rely on the theory of self-normalized processes [ 33 , 1 ] and block martingale conditions [ 40 ] . 3 Rebounding Bandits Problem Setup . Consider the set of K arms [ K ] : = { 1 , . . . , K } with bounded base rewards b1 , . . . , bK . Given a horizon T , a policy π1 : T : = ( π1 , . . . , πT ) is a sequence of actions , where πt ∈ [ K ] depends on past actions and observed rewards . For any arm k ∈ [ K ] , we denote its pull history from time 0 to T as the binary sequence uk,0 : T : = ( uk,0 , . . . , uk , T ) , where uk,0 = 0 and for t ∈ [ T ] , uk , t = 1 if πt = k and uk , t = 0 otherwise . The subsequence of uk,0 : T from t1 to t2 ( including both endpoints ) is denoted by uk , t1 : t2 . At time t , each arm k has a satiation level sk , t that depends on a satiation retention factor γk ∈ [ 0 , 1 ) , as follows sk , t : = γk ( sk , t−1 + uk , t−1 ) + zk , t−1 , ∀t > tk0 , ( 1 ) where tk0 : = mint { t : uk , t = 1 } is the first time arm k is pulled and zk , t−1 is independent and identically distributed noise drawn from N ( 0 , σ2z ) , accounting for incidental ( uncorrelated ) factors in the satiation dynamics . Because satiation requires exposure , arms only begin to have nonzero satiation levels after their first pull , i.e. , sk,0 = . . . = sk , tk0 = 0 . At time t ∈ [ T ] , if arm k is played with a current satiation level sk , t , the agent receives reward µk , t : = bk − λksk , t , where bk is the base reward for arm k and λk ≥ 0 is a bounded exposure influence factor . We use satiation influence to denote the product of the exposure influence factor λk and the satiation level sk , t . In Figure 1 , we show how rewards evolve in response to both pulls and the stochastic dynamics under two sets of parameters . The expected reward of arm k ( where the expectation is taken over all noises associated with the arm ) monotonically decreases by diminishing amounts with consecutive pulls and increases with disuse by diminishing amounts . Remark 1 . We note that there exist choices of bk , γk , λk for which the expected reward of arm k can be negative . In the traditional bandits setup , one must pull an arm at every time step . Thus , what matters are the relative rewards and the problem is mathematically identical , regardless of whether the expected rewards range from −10 to 0 or 0 to 10 . In addition , one might construct settings where negative expected rewards are reasonable . For example , when one of the arms corresponds to no recommendation with 0 being its expected reward ( e.g. , bk = 0 , λk = 0 ) , then the interpretation of negative expected reward would be that pulling ( recommending ) the corresponding arm ( item ) is less preferred relative to not pulling ( no recommendation ) . Given horizon T ≥ 1 , we seek a pull sequence π1 : T , where πt depends on past rewards and actions ( π1 , µπ1,1 , . . . , πt−1 , µπt−1 , t−1 ) , that maximizes the expected cumulative reward : GT ( π1 : T ) : = E [ ∑T t=1 µπt , t ] . ( 2 ) Additional Notation Let γ : = maxk∈ [ K ] γk and λ : = maxk∈ [ K ] λk . We use a . b when a ≤ Cb for some positive constant C . 4 Planning with Known Dynamics . Before we can hope to learn an optimal policy with unknown stochastic dynamics , we need to establish a procedure for planning when the satiation retention factors , exposure influence factors , and base rewards are known . We begin by presenting several planning strategies and analyzing them under deterministic dynamics , where the past pulls exactly determine each arm ’ s satiation level , i.e. , sk , t = γk ( sk , t−1 + uk , t−1 ) , ∀t > tk0 . With some abuse of notation , at time t ≥ 2 , given a pull sequence uk,0 : t−1 , we can express the satiation and the expected1 reward of each arm as sk , t ( uk,0 : t−1 ) = γk ( sk , t−1 + uk , t−1 ) = γk ( γk ( sk , t−2 + uk , t−2 ) ) + γkuk , t−1 = ∑t−1 i=1 γ t−i k uk , i , µk , t ( uk,0 : t−1 ) = bk − λk ( ∑t−1 i=1 γ t−i k uk , i ) . ( 3 ) At time t = 1 , we have that sk,1 ( uk,0:0 ) = 0 and µk,1 ( uk,0:0 ) = bk for all k ∈ [ K ] . Since the arm parameters { λk , γk , bk } Kk=1 are known , our goal ( 2 ) simplifies to finding a pull sequence that solves the following bilinear integer program : max uk , t K∑ k=1 T∑ t=1 uk , t ( bk − λk t−1∑ i=0 γt−ik uk , i ) : K∑ k=1 uk , t = 1 , ∀t ∈ [ T ] , uk , t ∈ { 0 , 1 } , uk,0 = 0 , ∀k ∈ [ K ] , ∀t ∈ [ T ] ( 4 ) where the objective maximizes the expected cumulative reward associated with the pull sequence and the constraints ensure that at each time period we pull exactly one arm . Note that ( 4 ) includes products of decision variables uk , t leading to bilinear terms in the objective . In Appendix A , we provide an equivalent integer linear program .

Summary: This paper introduces a problem of modeling satiation effects in multi-armed bandits. For example, if we assume the arms are restaurants that an agent is going to over time. Said agent may have a preference for pizza hut, but eating there every night would lead to a lower reward than it would if you haven't been to pizza hut in a while. The authors prove that the greedy algorithm is optimal when satiation retention, base rewards, and satiation influence is known and is equivalent over all arms. The authors then provide an upper bound on regret when using their EEP (Explore, Estimate, Plan) policy. The authors also provide empirical results on a toy version of the problem.