Rebounding Bandits for Modeling Satiation Effects

1 Introduction . Recommender systems suggest such diverse items as music , news , restaurants , and even job candidates . Practitioners hope that by leveraging historical interactions , they might provide services better aligned with their users ’ preferences . However , despite their ubiquity in application , the dominant learning framework suffers several conceptual gaps that can result in misalignment between machine behavior and human preferences . For example , because human preferences are seldom directly observed , these systems are typically trained on the available observational data ( e.g. , purchases , ratings , or clicks ) with the objective of predicting customer behavior [ 4 , 27 ] . Problematically , such observations tend to be confounded ( reflecting exposure bias due to the current recommender system ) and subject to censoring ( e.g. , users with strong opinions are more likely to write reviews ) [ 41 , 16 ] . Even if we could directly observe the utility experienced by each user , we might expect it to depend , in part , on the history of past items consumed . For example , consider the task of automated ( music ) playlisting . As a user is made to listen to the same song over and over again , we might expect that the utility derived from each consecutive listen would decline [ 35 ] . However , after listening to other music for some time , we might expect the utility associated with that song to bounce back towards its baseline level . Similarly , a diner served pizza for lunch might feel diminished pleasure upon eating pizza again for dinner . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The psychology literature on satiation formalizes the idea that enjoyment depends not only on one ’ s intrinsic preference for a given product but also on the sequence of previous exposures and the time between them [ 3 , 6 ] . Research on satiation dates to the 1960s ( if not earlier ) with early studies addressing brand loyalty [ 42 , 28 ] . Interestingly , even after controlling for marketing variables like price , product design , promotion , etc. , researchers still observe brand-switching behavior in consumers . Such behavior , referred as variety seeking , has often been explained as a consequence of utility associated with the change itself [ 25 , 17 ] . For a comprehensive review on hedonic decline caused by repeated exposure to a stimulus , we refer the readers to [ 11 ] . In this paper , we introduce rebounding bandits , a multi-armed bandits ( MABs ) [ 37 ] framework that models satiation via linear dynamical systems . While traditional MABs draw rewards from fixed but unknown distributions , rebounding bandits allow each arm ’ s rewards to evolve as a function of both the per-arm characteristics ( susceptibility to satiation and speed of rebounding ) and the historical pulls ( e.g. , past recommendations ) . In rebounding bandits , even if the dynamics are known and deterministic , selecting the optimal sequence of T arms to play requires planning in a Markov decision process ( MDP ) whose state space scales exponentially in the horizon T . When the satiation dynamics are known and stochastic , the states are only partially observable , since the satiation of each arm evolves with ( unobserved ) stochastic noises between pulls . And when the satiation dynamics are unknown , learning requires that we identify a stochastic dynamical system . We propose Explore-Estimate-Plan ( EEP ) an algorithm that ( i ) collects data by pulling each arm repeatedly , ( ii ) estimates the dynamics using this dataset ; and ( iii ) plans using the estimated parameters . We provide guarantees for our estimators in § 6.2 and bound EEP ’ s regret in § 6.3 . Our main contributions are : ( i ) the rebounding bandits problem ( §3 ) , ( ii ) analysis showing that when arms share rewards and ( deterministic ) dynamics , the optimal policy pulls arms cyclically , exhibiting variety-seeking behavior ( §4.1 ) ; ( iii ) an estimator ( for learning the satiation dynamics ) along with a sample complexity bound for identifying an affine dynamical system using a single trajectory of data ( §6.2 ) ; ( iv ) EEP , an algorithm for learning with unknown stochastic dynamics that achieves sublinear w-step lookahead regret [ 34 ] ( §6 ) ; and ( v ) experiments demonstrating EEP ’ s efficacy ( §7 ) . 2 Related Work . Satiation effects have been addressed by such diverse disciplines as psychology , marketing , operations research , and recommendation systems . In the psychology and marketing literatures , satiation has been proposed as an explanation for variety-seeking consumer behavior [ 11 , 25 , 26 ] . In operations research , addressing continuous consumption decisions , [ 3 ] propose a deterministic linear dynamical system to model satiation effects . In the recommendation systems community , researchers have used semi-Markov models to explicitly model two states : ( i ) sensitization—where the user is highly interested in the product ; and ( ii ) boredom—where the user is not engaged [ 18 ] . The bandits literature has proposed a variety of extensions where rewards depend on past exposures , both to address satiation and other phenomena . [ 14 , 21 , 39 ] tackle settings where each arm ’ s expected reward grows ( or shrinks ) monotonically in the number of pulls . By contrast , [ 19 , 2 , 7 ] propose models where rewards increase as a function of the time elapsed since the last pull . [ 34 ] model the expected reward as a function of the time since the last pull drawn from a Gaussian Process with known kernel . [ 43 ] propose a model where rewards are linear functions of the recent history of actions and [ 29 ] model the reward as a function of a context that evolves according to known deterministic dynamics . In rested bandits [ 12 ] , an arm ’ s expected rewards change only when it is played , and in restless bandits [ 44 ] rewards evolve independently from the play of each arm . Key Differences This may be the first bandits paper to model evolving rewards through continuousstate linear stochastic dynamical systems with unknown parameters . Our framework captures several important aspects of satiation : rewards decline by diminishing amounts with consecutive pulls and rebound towards the baseline with disuse . Unlike models that depend only on fixed windows or the time since the last pull , our model expresses satiation more organically as a quantity that evolves according to stochastic dynamics . To estimate the reward dynamics , we leverage recent advances in the identification of linear dynamical systems [ 40 , 38 ] that rely on the theory of self-normalized processes [ 33 , 1 ] and block martingale conditions [ 40 ] . 3 Rebounding Bandits Problem Setup . Consider the set of K arms [ K ] : = { 1 , . . . , K } with bounded base rewards b1 , . . . , bK . Given a horizon T , a policy π1 : T : = ( π1 , . . . , πT ) is a sequence of actions , where πt ∈ [ K ] depends on past actions and observed rewards . For any arm k ∈ [ K ] , we denote its pull history from time 0 to T as the binary sequence uk,0 : T : = ( uk,0 , . . . , uk , T ) , where uk,0 = 0 and for t ∈ [ T ] , uk , t = 1 if πt = k and uk , t = 0 otherwise . The subsequence of uk,0 : T from t1 to t2 ( including both endpoints ) is denoted by uk , t1 : t2 . At time t , each arm k has a satiation level sk , t that depends on a satiation retention factor γk ∈ [ 0 , 1 ) , as follows sk , t : = γk ( sk , t−1 + uk , t−1 ) + zk , t−1 , ∀t > tk0 , ( 1 ) where tk0 : = mint { t : uk , t = 1 } is the first time arm k is pulled and zk , t−1 is independent and identically distributed noise drawn from N ( 0 , σ2z ) , accounting for incidental ( uncorrelated ) factors in the satiation dynamics . Because satiation requires exposure , arms only begin to have nonzero satiation levels after their first pull , i.e. , sk,0 = . . . = sk , tk0 = 0 . At time t ∈ [ T ] , if arm k is played with a current satiation level sk , t , the agent receives reward µk , t : = bk − λksk , t , where bk is the base reward for arm k and λk ≥ 0 is a bounded exposure influence factor . We use satiation influence to denote the product of the exposure influence factor λk and the satiation level sk , t . In Figure 1 , we show how rewards evolve in response to both pulls and the stochastic dynamics under two sets of parameters . The expected reward of arm k ( where the expectation is taken over all noises associated with the arm ) monotonically decreases by diminishing amounts with consecutive pulls and increases with disuse by diminishing amounts . Remark 1 . We note that there exist choices of bk , γk , λk for which the expected reward of arm k can be negative . In the traditional bandits setup , one must pull an arm at every time step . Thus , what matters are the relative rewards and the problem is mathematically identical , regardless of whether the expected rewards range from −10 to 0 or 0 to 10 . In addition , one might construct settings where negative expected rewards are reasonable . For example , when one of the arms corresponds to no recommendation with 0 being its expected reward ( e.g. , bk = 0 , λk = 0 ) , then the interpretation of negative expected reward would be that pulling ( recommending ) the corresponding arm ( item ) is less preferred relative to not pulling ( no recommendation ) . Given horizon T ≥ 1 , we seek a pull sequence π1 : T , where πt depends on past rewards and actions ( π1 , µπ1,1 , . . . , πt−1 , µπt−1 , t−1 ) , that maximizes the expected cumulative reward : GT ( π1 : T ) : = E [ ∑T t=1 µπt , t ] . ( 2 ) Additional Notation Let γ : = maxk∈ [ K ] γk and λ : = maxk∈ [ K ] λk . We use a . b when a ≤ Cb for some positive constant C . 4 Planning with Known Dynamics . Before we can hope to learn an optimal policy with unknown stochastic dynamics , we need to establish a procedure for planning when the satiation retention factors , exposure influence factors , and base rewards are known . We begin by presenting several planning strategies and analyzing them under deterministic dynamics , where the past pulls exactly determine each arm ’ s satiation level , i.e. , sk , t = γk ( sk , t−1 + uk , t−1 ) , ∀t > tk0 . With some abuse of notation , at time t ≥ 2 , given a pull sequence uk,0 : t−1 , we can express the satiation and the expected1 reward of each arm as sk , t ( uk,0 : t−1 ) = γk ( sk , t−1 + uk , t−1 ) = γk ( γk ( sk , t−2 + uk , t−2 ) ) + γkuk , t−1 = ∑t−1 i=1 γ t−i k uk , i , µk , t ( uk,0 : t−1 ) = bk − λk ( ∑t−1 i=1 γ t−i k uk , i ) . ( 3 ) At time t = 1 , we have that sk,1 ( uk,0:0 ) = 0 and µk,1 ( uk,0:0 ) = bk for all k ∈ [ K ] . Since the arm parameters { λk , γk , bk } Kk=1 are known , our goal ( 2 ) simplifies to finding a pull sequence that solves the following bilinear integer program : max uk , t K∑ k=1 T∑ t=1 uk , t ( bk − λk t−1∑ i=0 γt−ik uk , i ) : K∑ k=1 uk , t = 1 , ∀t ∈ [ T ] , uk , t ∈ { 0 , 1 } , uk,0 = 0 , ∀k ∈ [ K ] , ∀t ∈ [ T ] ( 4 ) where the objective maximizes the expected cumulative reward associated with the pull sequence and the constraints ensure that at each time period we pull exactly one arm . Note that ( 4 ) includes products of decision variables uk , t leading to bilinear terms in the objective . In Appendix A , we provide an equivalent integer linear program .

Summary: The paper proposes a Multi-armed Bandits model with non-stationary reward affected by users' satiation level. The authors consider two cases of satiation dynamic: known deterministic dynamics and unknown stochastic dynamics. Then the authors show that under specific conditions a greedy algorithm is optimal. In the general setting (unknown stochastic satiation dynamics), they propose a two-step algorithm EEP with error estimation and analysis.