Learning to Take a Break: Sustainable Optimization of Long-Term User Engagement
Authors: Eden Saig
Nir Rosenfeld
Venue: Submitted to ICLR 2023 (2023)
Keywords: Lotka-Volterra dynamics
breaking policies
digital well-being
feed-based recommendation

Abstract: Optimizing user engagement is a key goal for modern recommendation systems, but blindly pushing users towards increased consumption risks burn-out, churn, or even addictive habits. To promote digital well-being, most platforms now offer a service that periodically prompts users to take a break. These, however, must be set up manually, and so may be suboptimal for both users and the system.
In this paper, we propose a framework for optimizing long-term engagement by learning individualized breaking policies. Using Lotka-Volterra dynamics, we model users as acting based on two balancing latent states: drive, and interest---which must be conserved. We then give an efficient learning algorithm, provide theoretical guarantees, and empirically evaluate its performance on semi-synthetic data.

Under review as a conference paper at ICLR 2023
LEARNING TO TAKE A BREAK: SUSTAINABLE OPTIMIZATION OF LONG-TERM USER ENGAGEMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Optimizing user engagement key goal for modern recommendation systems, but blindly pushing users towards increased consumption risks burn-out, churn, or even addictive habits. To promote digital well-being, most platforms now offer a service that periodically prompts users to take break. These, however, must be set up manually, and so may be suboptimal for both users and the system. In this paper, we propose a framework for optimizing long-term engagement by learning individualized breaking policies. Using Lotka- Volterra dynamics, we model users as acting based on two balancing latent states: drive, and iiterest-whice must be conserved. We then give an efficient learning algorithm, provide theoretical guarantees, and empirically evaluate its performance on semi-synthetic data.
INTRODUCTION
As consumers of content, we have come to rely extensively on algorithmic recommendations. This has made the task of recommending- success of modern media platforms. Most commercial systems are built with the primary aim of optimizing user engagement, a process in which machine learning plays a central role. But alongside their many successes, recommendation systems have also been scrutinized for heedlessly driving users towards excessive and often undesired levels of consumption. This has raised awareness as to the need for redesigning recommendation systems in ways that actively promote digital well-being.
relevant, timely, and personalized manner- key to the
How can media platforms balance business goals with user well-being? One prominent approach, which is now offered by most major platforms, is to periodically prompt users to take breaks (Con- stine, 2018; Perez, 2018). The idea behind breaks is that occasional disruptions curb the inertial urge for perpetual consumption, and can therefore aid in reducing 'mindless scrolling' (Rauch, 2018), or even addiction (Montag et al., 2018; Ding et al., 2016). As a general means for promoting well-being, breaking is psychologically well-grounded (e.g., Danziger et al., 2011; Sievertsen al., 2016). But for platforms, breaks serve a utilitarian purpose: their goal is to foster long-term engage- ment by compensating for the myopic nature of conventional recommendation algorithms, which are typically trained to optimize immediate engagement. Since breaking schedules are applied heuris- tically on top of existing recommendation policies-and typically need be set up manually by users current solutions unlikely utilize their full potential (Monge Roffarello & De Russis, 2019).
In this paper, we propose a disciplined learning framework for responsible and sustainable optimiza- tion of long-term user engagement by controlling breaks. Our point of departure is that sustained engagement necessitates sustained user well-being, and here we advocate for breaks as a means to establish both. Focusing on feed-baser recommendation, our framework optimizes long-term en- gagement by learning an optimal breaking policy that prescribes individualized breaking schedules.
The challenge in learning to break is that the effects of recommendations on users can slowly accu- mulate over time, deeming as ineffectual policies that rely on clear signs of over-exposure, To be preemptive, we argue that breaks must be scheduled in a way that anticipates the future trajectory of user behavior, and early on. To achieve this, we introduce novel class of behavioral models based on Lolta-Volterra (LV) dynamical systems (Lotka, 1910). These depict users as acting based on two balancing forces: drive to consume and intrinsic interest, with corresponding latent states. Intuitively, high interest increases drive consume, but prolonged consumption decreases interest; together, these describe how user behavior varies over time and in response to recommendations. Our model captures the notion that interest can exhaust long before over-consumption is observed.

---

Under review as a conference paper at ICLR 2023
This arms approach with the prescience needed to prevent burn-out by ensuring that interest is sustainably preserved; thus, whereas current solutions target the symptom curpoom-ours ours aims for the cause.
Our proposed learning algorithm consists of two steps: First, embed user interaction sequences in 'LV-space' -arraceeehhheee set of all possible trajectories that our behavioral model class can express. Then, optimize individualized breaking policies by solving an optimal control problem over this la- tent space, in which the control variable is a breaking schedule applied on top of the existing rec- ommendation scheme. Here the challenge is that different breaking policies can lead to different counterfactual trajectories, of which observational data is only partially informative. Since our goal considers long-ten outcomes, our solution to optimize directly for counterfactual steady-states. From a behavioral perspective, we view this as aiming to steer towards sustainable habits; from a computational perspective, under our choice of policy class, this enables tractable learning.
As we show, the optimization landscape of LV equilibria admits compact representation, whose main benefit is that it can be fully described by predictions of individualized user engagement rates. Practically, this is advantageous, as it circumvents the need to take arbitrary and costly exploration steps, and enables learning using readily available predictive tools (e.g., Gupta et al., 2006). We make use of a small set of learned predictive models, each trained on small and minimally-invasin experimental dataset, which allow us to tune our policy to suit different conditions. The final learned policy has an intuitive interpretation: it takes imput small set of predictions for user, and via careful interpolation, applies a decision rule that anticipates the effects of breaking on future out- comes (c.f. conventional approaches, which take in predictions and apply the myopic argmax rule).
Our main theoretical result is a bound on the expected long-term engagement of our learned breaking policy, relative to the optimal policy in the class. We show that the gap decomposes into three distinct additive terms: (i) predictive error, (ii) modeling error (i.e., embedding distortion). and (iii) variance around the (theoretical) steady state. These provide an intuitive interpretation of the bound, as well as means to understand the effects of diffcrent modeling choices. Our proof technique relies on carefully weaving LV equilibrium analysis within conventional concentration bounds for learning.
Finally, we provide an empirical evaluation of our approach on semi-synthetic data. Using the MoiveLens 1M dataset, we generate discrete time-series data in a way that captures the essence of our behavioral model, but is different from the actual continuous Cantinuous-time dynamics we optimize over. Results show that despite this gap, our approach improves significantly over myopic baselines, and often closely matches an optimal oracle. Taken together, these demonstrate the potential utility of our approach. Code is available at: https://github..oom wwww/wollmi//ooooooooccc caml-iccr-2000 2023/2003/l
Broader perspective. At a high level, our work argues for viewing recommendation as a task of sustainuble resource management. As other cognitive tasks, engaging with digital content requires the availability of certain cognitive resources resources-atteniioas attentional, executive, or emotional. These resources inherently limited, and prolonged engagement depletes them (Kahneman, 1973; Muraven & Baumeister, 2000); this, in turn, can reduce the capacity of key cognitive processes (e.g.. perception, attention, memory, self-control, and decision-making: and in the extreme cause ego depletion (Baumeister et al., 1998) or cognitive fatigue Mullette-Gillman et al., 2015). As a means to allow resources to replenish, 'mental breaks' have been shown be highly effective (Bergum & Lehr, 1962; Hennfig et al., 1989; Gilboa et al., 2008; Ross et al., 2014; Helton & Russell, 2017).
Nevertheless, traditional approaches recommendation remain agnostic to the idea that recom- mending takes a cognitive toll: they simply recommend at each point in time the item predicted to be most engaging (Robertson, 1977). As an alternative, our approach explicitly models recommen- dation as a process which draws on these resources, and therefore- mustt also conserve them. The subclass of 'Predator-Prey' LV dynamics which we draw on are used extensively in ecology for mod- eling the dynamics of interacting populations, and demonstrate how over-predate wver-predttion can ultimately lead to self-extinction by eliminating the prey population-bov also show how enabling resources to naturally replenish ensures sustainable relations. As such, here we advocate for studying recom- mendation systems as numan-ccntrr ecosystems, and take one step towards their sustainable design.
1.1 RELATED WORK
User dynamics: latent states and feedback. A recent body of work aims to capture time- varying behavior by modeling users as acting based on dynamic latent states. Broadly, works in this field
2

---

Under review as a conference paper at ICLR 2023
model the effects of recommendations either shifting user preferences via positive-only feedback (Jiang et al., 2019; Kalimeris et al., 2021; Sanna Passino et al., 2021; Dean & Morgenstern, 2022). or reducing willingness to consume via negative-only feedback, e.g. via boredom, satiation, fatigue (Wang & Lin, 2003; Warlop et al., 2018; Kleinberg & Immorlica, 2018; Cao et al., 2020; Leqi et al., 2021). While these restrict behavior expressing unidirectional effect, our approach integrates both types of feedback and models internal states as competing but balancing forces, which we argue more realistic. This draws connections recent attempts of injecting psychological modeling into recommendation system design (Kleinberg et al., 2022; Dubey et al., 2022; Curmei et al., 2022). Works in this field often combine theoretical analysis with simulation studies (Schmit & Riquelme, 2018; 8; Chaney et al., 2018; Mansoury et al., 2020; Krauth et al., 2020), and here we follow suit.
Volta-Votteraaaaaoo dynamics: modeling, learning, and control. The study of ecosystem dynamics and their conservation has a long and rich history, in which LV analysis plays an integral role (see e.g. Hofbauer et al. (1998); Takeuchi (1996)). LV systems our used primarily for modeling biologi- cal ecosystems, but are also used in economies (Weibull, 1997; Samuelson, 1998), finance (Farmer, 2002; Scholl et al., 2021), and behavioral modeling (e.g., drug addiction and relapse (Duncan et al., 2019)). In terms of learning, Gorbach et al. (2017) and Ryder al. (2018) propose variational tech- niques for dynamical systems, but do not consider control. Our work aims to directly learn optimal policies. for which we draw on recent advances in turnpike optimal control (Trélat & Zuazua, 2015).
LEARNING SETTING
We consider a sequential recommendation setting in which users interact with stream of recom- mended items over time. New users u are sampled independently from some unknown distribution D, and time for each user is measured relative to their time of joining. Interactions occur discrete time-points in continuous time, / E R4, and upon user request: when user queries the system for additional content at time L, the system responds by presenting a recommended item xx(1) E X. We assume recommendations are governed by existing and fixed recommendatton policy s), which determines x(t) given a request from u at time t. We allow & to be stochastic, and make no ad- ditional assumptions on its structure mechanics. User interactions are therefore described by sequence of pairs (11, x:)); where 24 = xtt,) ~ (b(u;44) E X' is the item recommended to u at time f,.
Our key modeling assumption is that subsequent request times, 4,41 = t, + At, for At, > O, are determined jointly by the user's 'state" time 4 and the recommended item æ; (note this means Al; can depend on 42). We consider user states as latent and in the abstract, but broadly expect 'good' recommendations
entail frequent requests by inducing small values of At,. Together, user u's choice behavior, coupled with the policy vi, induce temporal point process (TPP) which governs the generation of interaction sequences of duration t as ((t,,r:) | f E [0, t]} ~ TP:
For each user, the system collects data until some (relative) fixed time To. and seeks to optimize engagement in the subsequent interval [To, To + T], where T is a predetermined time horizon. We denote the corresponding input sequences by 5
{(4,, 1/ E [0, and target sequences by S., = {(1,, axa) 44 E [To. T]}- Defining by 7 1|S the engugement rate of for the chosen time horizon T, our goal in learning will be to maximize expected long-term user engagement rate, namely E E + Sul] for (S S.) ~ TPP To + TT), in expectation over new users.
Breaking policies. The unique aspect of our learning problem is that our only means for increasing engagement is by suggesting breaks. Our task will be to learn a breaking policy E II which can override t: when user u queries the system at time t,, the policy πt л(.;; t,) {0, 1} determines whether to present the intended item (t) ~ (b t.) (for π 0). or suggest a break instead (π = 1).' Denoting the overall composed policy by π v/, our aim is learn a breaking policy I that increases engagement by complementing an existing qb. Hence, our learning objective is:
arg max Eu ETPP [+ [S]]],
(5) ~ TPP moy (w; To + T)
(1)
Broadly, we expect breaks to negatively affect short-term engagement (i.c., entail longer ensuing At;), but have the potential to improve engagement in the long run if scheduled appropriately.
This is similar in spirit to the 'learning defer' paradigm (Madras et al., 2018), but in a different context.
3

---

Under review as a conference paper at ICLR 2023
Data and exploration. For learning optimal breaking policy, we assume the system has access to dataset of previously logged interactions S = {(SP. Si)...., (S Sm) for m train-time users u1...., Um D. collected under 'clean' recommendation policy s). It will be convenient to assume that the system "featurizes" user input sequences S via vector mapping & used in learning. For clarity, we overload notation and represent users as ((S Rd This allows us support additional user feedback (e.g., ratings) or information (e.g., demographics) as input to Φ.
Since Eq. (1) is policy problem (note breaks affect outcomes), learning requires some form of exploration or experimentation. Here we aim for experimentation to be simple and minimal. Specif- ically, we assume access to a small number of N additional datasets, $D) = {(ux, [x)] col- lected for different users, and under composed policies マj for various predetermined ÿt These datasets can either be given at the onset, or collected as part of the learning process; our only usage of them will be for learning predictors D = fu) of engagement rate y + Ideally, we would like to make do with only a few 8(5) of small size and that can be gathered concurrently; our results show that even a single additional dataset can be highly informative, and in some cases sufficient.
Learning to break. To effectively optimize Eq. (I), learning must anticipate the effects of different breaking policies π E II on future sequences S, of unobserved users u. It will therefore be useful to distinguish between the policy π itself, which determines when breaks are applied, and component for estimating individualized counterfactual engagement rates + (T)) induced by オ, which will aid choosing a good policy. Our focus will be on learning simple policies coupled with rich and task-appropriate models of responsive user behavior. In particular, we set II to include all stationary policies, x (2) = (P=). which for each user u recommends a break with a time-independen sonalized probability P., p(u) E [0, 1]. Stationary policies are interpretable, amenable to efficient optimization, and straightforward to implement. As we show, despite their simplicity, such policies can be quite expressive when coupled with an appropriate behavioral model.
per-
ENGAGEMENT VIA 'PREDATOR-PREY DYNAMICS
To optimize engagement with breaks, we propose to model user behavior as a dynamical system of Lotka- Volterra (LV) Predator-Prey equations. The model operates over continuous dynamics, which has analytic and optimizational benefits; we later make the connection back to discrete sequences.
Behavioral model. Our model of user behavior views each user acting based two types of time-dependent latent variables: internal drive for consumption, denoted by A(t) E IR,;; and intrinsic interest, denoted by q(t) [0, 1]. Intuitively, we expect that engaging experiences will reinforce user's desire to further consume; conversely, excessive exposure to content should slowly 'erode" her interest-butt but left alone, will allow it to replenish (Thoman et al., 2011). Thus, drive and interest act as balancing forces. Together, we model the time-dependent relations between X(t) and q(t) as:
dp di
dg
= ax + B3g,
79(1 - 9) -8x di
(2)
6) > 0 parameterize the dynamics. For À (drive, or 'predator"), determines its where 0 (a. B, 7,
natural decay rate, and / its interest-dependent self reifforiffrre rate; for (interest, or 'prey'), "), ? specifies its natural replenishment rate, and 8 its rate of depletion from consumption. Note the two equations are coupled: q mediates the reinforcement of X, and À mediates the depletion of q.
Dynamical properties. The LV model in Eq. (2) describes consumption as a cycle: when interest q(() is high, drive to consume X(l) increases, resulting in positive feedback; conversely, when X(I) is high, (t) decreases, which expresses negative feedback. In general, À grows until interest is too low to sustain consumption, which point consumption drops sharply, allowing interest to recover-an and the cycle repeats. The cycling behavior exhibits oscillations in À and q. with one lagging after the other. A typical trajectory is illustrated in Figure 1. Note how the drop in À occurs only some time after q is depleted; hence, anticipating (and preventing) the collapse of À requires knowledge (and conservation) of q. Thus, 9 serves as a resource: necessary for consumption, and of limited supply.
Over time, and if no interventions are applied, the magnitude of oscillations decreases, and the system naturally approaches stable equilibrium, denoted (X²,q"), ,""), determined by the system pa-
4

---

Under review as a conference paper at ICLR 2023
A(E) and g(t) - Joint evolution over time
(A.g) Phase space diagram
Consumption drive X(O)
(200-400))
Consumpioon dinee A(
Intrinsic g(f)
0.7
.....: Cqulibrium concumption X
10-
0.6
of
Interest o(
0.6
10-
9.5
0.3
20
00
90
100
10
15
20
Time
Comsumption
Figure 1: Characteristic LV dynamics (Bq. (2)). (Left) Temporal relations between consumption drive X(), intrinsic interest q(!), and equilibrium X². Note how drops only some time after q has depleted. (Right) Evolution of the same system in phase space (A, q). with its equilibrium (X*, gg").
rameters 0 (a, B, 7,,6) 8) and which attracts all initial conditions A(0), g(0) (Takeuchi, 1996). Equi- librium plays a key role in our approach, as it captures the notion of habits, which we aim to improve.
Engagement maximization as optimal control. To optimize engagement (Eq. (I)), we propose to optimize X as an alternative proxy. Broadly, our approach will be to associate with each user u a dynamical system parameterized by 0n = (an, Bu: Ya: 6n), and recommend breaks which lead to high values of the corresponding X = Xg..- Intuitively, if we think of IS. as 'empirical" rate (determined by the At; in Sw), then X; is its continuous theoretical counterpart, and so ideally would like to find a Ow for which X; is the limiting behavior of #1S., (i.e., when At - 0 and T - o.). In practice, we expect X to be useful target when the observed +|S exhibits habits that are 'close enough' to the theoretical equilibrium, and in Sec. 4.3 we make this condition precise.
Since our goal is to optimize a breaking policy, we must also be precise about the way breaks affect dynamics in our model. Towards this, we introduce into Eq. (2) control variable, p(t) E [0, 1]. which acts as a 'gate' that controls the mediation strength between A(t) and q(t) via:
dd
dg dt
= -a + 3Aq(1 - p),
70(1 - 9) 55g(1 pp)
(3)
dt
When p > O, breaking has a dual effect: it decelerates drive X, and at the same time, lets recover.
Our goal will now be to solve the optimal control problem of finding p(t) which maximizes X" Note that not every dynamic controlling schedule guarantees convergence to some X Hence, we focus on fixed controls, p [0, 1]. which we prove converge to equilibrium, and having closed form. 7: 8) define an LV system as in Eq. (3). Then for any E [0, 1], we have: Lemma 1. Let = (a, 3,7,
7 1 8 81-p
a 11 31-p)
(p)
if p e (0, a/š), and zero otherwise.
(4)
Proof in Appendix A. For optimization, Lemma 1 is useful since it depicts X as function of p, parameterized by 8. Thus, given some 8ú for user u, our objective is solve the control problem:
Pu- = arg max pe No. (p)
(5)
Optimizing engagement now reduces to solving Eq. (5); given Pu. we make the connection back to discrete time by setting the breaking policy to be π (p.), which recommends breaks rate Pu.
4 LEARNING OPTIMAI. BREAKING POLICIES
We now turn to presenting our learning algorithm. Our approach to optimizing engagement consists of two steps: (i) associating with each user u a set of LV parameters 9u. and then (ii) finding Pu. which add intermediate prediction step, which allows us to 'shortcut' directly from observations to optimal policies. We conclude with anal- ysis showing when learned policies s ((p.) lead to good expected engagement E p Errr [+[S.]].
maximizes Xg. (p), and plugging into (Pp)). In practice,
5

---

Under review as a conference paper at ICLR 2023
4.1 EMBEDDING USERS IN LV SPACE
To find Ow. a seemingly reasonable approach would be to fit an LV trajectory to the initial sequence Sy. i.e., by solving ming E; |L >4(4) for the observed E S.. This be interpreted as embedding Sh in "LV-space" by finding the nearest continuous trajectory, for which 0 provides compact representation. However, a key issue with this approach is that 5h contains past observa- tions made under single policy π (p)) (e.g., the 'clean' policy x (0)). To optimize future engagement, the learned O,, must correctly account for the affects of general on possbile ensuing sequences S..
As alternative, we propose find On by fitting the entire equilibrium curve of (p) (Eq. (4)). Formally, for each p, define the expected empirical engagement rate Xa (p) as:
(p) E., [+
S. 3h~TPP TPP =(p) (u:
(6)
i.e. Ā is the rate of counterfactual future trajectories for all possible choices of p. Ideally, we would like to find 0 for which the learned curve (p) closely aligns with that of X (p) across p E [0, 1]:
0, arg arg min A Xb
8
for some function norm ||-||, and for which X and X have similar maximizing p (since we aim for optimizing Xg to work well for Xu). Unfortunately, Xy is theoretical construct, and so 6, cannot be obtained from observations. Hence, we propose to replace Xu with finite set of predictions.
The role of prediction. Recall our input consists of a primary dataset ((0) = S = {(u,, 5,))m
as well as additional 'experimental" datasets st) collected under different π, = π (p,) and of sizes my for j N. We can use these to learn individualized, policy-specific predictors fu( = pps (u), trained to predict for each user u her engagement rate y = |1.a1 under Ty. For ex- ample, if we train f, to minimize the squared error [x( - yı²² on pairs (ux, 9k) E 8 then fu) should be a reasonable estimator of the expected Au (p,). Hence, for given u, a finite set of pairs {(p,, fz (1 gives points to which we can fit @ to Xg. Our final criterion for choosing 6. is:
0, = arg min fu) - Abol = arg min
(1) - XT(,)))
(8)
1=1
given here with the lg vector norm, and where f(u) will see next, optimizing over p requires only the ratios a/B and 7/8, in which X is quadratic. Hence, Eq. (8) can be efficiently solved using a polynomial Non-Negative Least Squares (NNLS) regression solver (Chen & Plemmons, 2010).
E. As we
The role of experimentation. Two parameters control the goodness of fit for 0.: the number of experimental datasets, N, and their sizes, my for j E [N]. In general, increasing N provides more data points for solving Eq. (8), and increasing each my reduces noise for that point (i.e., fu(p) should be closer to X.). But experimentation is costly, and so in reality N and the m, may be small. As motivation, we next show that under realizability and for accurate predictions, N = suffices.
Proposition Fix = 2, and let po: PI E [0, 10,1-a/ a/3). For a user u, (i) exists On S.1. Xn = No. and (ii) fi(u) Au (p:) for i 1, then 0,, is optimal, i.e., solving Eq. (8) gives 0., A..
Proof in Appendix A, and relies on Lemma 1. Next, we discuss how to obtain (u) from 0n,
4.2 FROM PREDICTIONS TO OPTIMAL POLICIES
Recall our goal is to learn an individualized breaking policy т that for each user u applies an appro- priate, personalized breaking schedule T (u). To be able to generalize to unseen u, the conventional approach is to learn a parameterized policy (2:; 7), where v) is optimized on training data, and ap- plied to new users test-time. Our approach, by relying on predictions, circumvents the need to learn parameterized policies: once the predictors {f, [f]) have been learned, the policy problem decomposes over users, and optimal policies (u) are determined independently for each user.

---

Under review as a conference paper at ICLR 2023
LV equilibrium consumption X1(p)
Optimal stationary policy 5 (a/9)
Policy optimization from data
0.7
0.7
Optinal palicy (e")
Cquibrium consumption
0.3
4-
Commupptoo
6-
2-
Ag(p) Ag. (P)
Ag. (P)
0.-
0.0
0.2
0.4
0.6
3
0.0
0.2
0.4
0.5
Probusility
purameters (121n)
Prolusiliity
Figure 2: (Left) Equilibrium curves (p) and optimal policies p" (markers) for user types (0..) that: benefit from breaks (green), do not require breaks (orange), and will inevitably churn (blue). (Center) The optimal policy p" for all 8/a, showing second-order phase change at B/a 2. (Right) An illustration of the true counterfactual engagement curve (orange).
Optimal policies. Given On. our next result establishes a closed form solution for the optimal pu- Note that Eq. (4) shows Xp(p) is piece-wist wise polynomial in 1/(1 p). Solving for get: Lemma 2. Let 0 (a, B, 7, 6) define an LV system as in Eq. (3). Then the optimal p" is given by:
p' (0) arg max x5(p) 1 if P
and zero otherwise. 2
Proof in Appendix A.1. Hence, once 0, is found, our learned policy is defined as:
Tu)) = (px), where pu p (6w)
(10)
Altogether, to compute Pu. the least squares approach in Eq. (8) applied to obtain the polynomial coefficients 2u 0o7a 3.8a ). then du is estimated from their ratio, and then plugged into Eq. (9) to ob- 3.
tain Pa.. For N = 2, (u) has a closed-form formulation function of predictions (Appendix A.2).
Additionally, note that by Eq. (10), P.. exhibits a phase transition at az B. = 2.2 below which p..
Corollary 1. In LV space, (u) partitions users to those who require breaks, and those who don't 't. Corollary 2. In the realizable case of Prop. x(pu) idempotently improves over the myopic x (0).
Thus, the optimal policy can be interpreted as suggesting breaks only when it deems them necessary. Figure illustrates X (p) curves, phase shift, and optimal and learned policies for various user types.
4.3 THRORETICAL GUARANTEES
We are now ready state our main theoretical result, which bounds the expected long-term engage- ment obtained by our learned policy, f.. Our bound shows that the gap between á and the optimal policy, opt is goverened by three additive terms, each relating to different aspect of our ap- proach: modeling error (Eiv), predictive error (Epred), and deviation from expected behavior (Edev). description and interpretation of each term follows shortly. For simplicity, we focus on = 2.
Theorem 1 (Informal). Let po, P1 E [0, 1). and denote by opt IT be the optimal stationary policy. Then for the learned breaking policy ât we have:
E Eu
[
E., [+ <
TrPe (ELV + Epred + Edev)
PP Pol
where Vroe is a speciccc constant scale factor.
Formal statement, precise definitions, and proof are given in Appendix A.3. The proof consists of three main steps: We begin with a clean LV system at T = oc, and quantify the downstream effect of perturbing the optimal policy. Then, plug in the learned policy, and bound the gap due to predic- tive errors and finite T. The final step makes the transition from continuous dynamics to the discrete TPP. We next detail the role of each of the five terms in the bound, and how they can be controlled.

---

Under review as a conference paper at ICLR 2023
Since targets y + [Sul are real, Epred is simply the expected regression error over users, measured here in RMSE. As is standard, Epred can be reduced by increasing the num- ber of samples m, or by learning more expressive predictive functions f (e.g.. larger neural nets).
Predictive error:
Modeling error: L.V dynamics permit tractable learning; but as any hypothesis class, this trades off with model capacity. Here, SLV quantifies the error due limited expressive power. Further re- ducing SLV can be achieved by consider richer dynamic models-a challenge left for future work.
Deviation from expectation: The learned f.. rely on predicted equilibrium, but are trained on finite-horizon data. In expectation, édev captures how finite sequences deviate from their mean. As a rule of thumb, we expect larger T to reduce this form of noise, but this cannot be guaranteed.
Sensitivity For N = 2, the term |p1 Pul quantifies the added value of exploring beyond the default breaking policy po- Intuitively, when the points are farther away, fitting the equilibrium curve is easier. Thus, for the likely case of Po 0, the experimental breaking pi should be chosen to balance between performance gain and overexposure of experimental subjects to breaks.
EXPERIMENTS
We conclude with an empirical evaluation of our approach on semi-synthetic data. We use the MovieLens 1M dataset to generate recommendations and simulate user behavior, enabling us to evaluate counterfactual outcomes under different policies. See Appendix B for additional details.
5.1 EXPERIMENTAL SETUP
Data. The MovieLens 1M dataset (Harper & Konstan, 2015) includes 1,000,209 ratings provided by 6,040 users and for 3,706 items, which we use to obtain features, determine the dynamies, and emulate é. We sample and hold out 30% of all ratings Tua via user-stratified sampling, to which we apply Collaborative Filtering (CF) to get user features u and item features æ that approximate u x R Fur (d = 8, RMSE 0.917, E 5).. This mimics process where á is based on items recom- mended by & and rated by users. We then take the remaining data points and randomly assign 1,000 users to the test set, which evaluate policies. The remaining users are assigned to the train set, which is then further partitioned into the main S and the different sti) per experimental condition.
Recommendation policy and user behavior. Following Kalimeris al. (2021); Hron al. (2022), we set tú to recommend based on learned softmax scores, softmax,, taken over all non held-out æ having ratings for u. For user behavior, we generate discrete interaction sequences S, in a way that relates to our dynamic model, but is nonetheless quite distinct. Specifically, each user is associated with discrete-tine latent-state variables, updated via an LV discretization scheme (see Appendix B.5). These, together with recommended items At, E S,,. Importantly. we let At, depend on u's rating for x,, namely Tu,,,, which we interpret as u's utility from consuming zi. We parameterize this dependence via 'immediate' tiem-specific consumption gain parameters But that depend on Tu,a:: and for simplicity set aw: Ya: 8, to be fixed.
zxfa), determine consecutive
Note that since B.s depends on the recommended æ;, it varies in time, and hence there is no single B that underlies the dynamics: even in the limit (At - 0, + oo), user behavior cannot be described by continuous LV system, which implies SIN > 0 (see Appendix Figure 5). Since the baseline RMSE is high, we set Bus x f2 u.a. where Fu.z, = KTa,z, + (1 k)u" x, so that x interpolates between predicted ratings (R 0) and true ratings (n = 1). This allows us (indirectly) control 6prod- For all experiments we use = 100, and so expect roughly fixed edev > 0.
Methods. We compare our approach (LV) to several baselines: (i) a default policy which my- opically optimizes for immediate engagement (and so does not break); (ii) 'safety switch' policy (safety0r) that breaks once consumption surpasses a threshold т: (iii) a prediction-based policy (best-of) that chooses the best observed Pv, arg max,, fj (w) (rather than optimizing over Pv. E [0, 1)); and (iv) an oracle benchmark which directly optimizes the (generally unknown) true under- lying dynamics. We measure mean long-temn engagement rate (LTE) for each approach, and report averages and standard errors computed over 10 random splits. Performance is measured relative to the default baseline it represents change in policy (typical absolute values are LTE xs 10).
8

---

Under review as a conference paper at ICLR 2023
Engagement for different policies (ix = (.5)
Engagement per LV breaking schedule
Engagement for varying Pi (Au
8%
% inppeveeennt (va. deaul))
20%6
decision
impervement dafuu)))
6%
6%
15%
47%
4%
10%
IV. <
2%
IV.
0
55%
best-of,
0%
0%
0%
bent-of, A 0.5
Jelsult
besi-dl,
-2%
-2%
dedail
4afesy814 1i
best-ff
Godd
0.0
0.1
11.2
0.3
0.4
0.1
002
0,3
0.4
policy (66)
Ansigned treatment (p:)
Figure 3: Results the MovieLenn 1M dataset. (Left) Performance gain of different approaches (relative to default policy). (Center) Performance of LV by user group, partitioned by learned policies P.. (Right) Sensitivity to increasingly aggressive experimental P1 (N 2, 0).
5.2 RESULTS AND ANALYSIS
Main results. Figure 3 (left) compares the performance of method to other policies. Here set O, use N = 3 with P5 {0.05, 0.1, 0.15). and consider = 0.5 (note x. affects all poli- cies). As can be seen, our approach significantly improves over default (+6.48%). For safety0r, 16 (+5.68%; chosen in hindsight) shows the importance of
improvement over the optimal being preemptive; for the slightly smaller v 14, breaks are harmful. The gap from best-of (+2.19%) quantifies the gain from the optimization step in Eq. (9), and the close performance to oracle (-0.918% suggests that optimizing the empirical curve (Eq, (8)) works well as as proxy.
User types. Figure 3 (center) shows for our approach how gain in LTE varies across learned break- ing policies p.. > 0. For increasingly- accuurate predictions (N E {0, 0.5, 1)), the main plot shows performance gains for each group of users, partitioned by their P.. values (binned; plot shows average and unit standard deviation per bin). Gains until p.. S 0.15 are mild, but for P.. >> 0.15, the general trend is positive: users who are deemed require more frequent breaks, benefit more from breaking. Gains until P.. S. 0.3 increase for all K, but for P. > 0.3, extrapolation becomes difficult: note the higher variation within each K, as well as significant differences across x. This highlights the impor- tance of accurate predictions for inferring optimal P.. when the experimental pt are small. The in- laied plot shows that, in line with our theory, P.. exhibits an empirical phase shift in the estimated 0,-
Treatments. Figure: (right) shows the effects of experimental treatments on performance. Fo- cusing N = 2, fix po = 0, and consider increasingly aggressive experimentation by varying P1 E 0.4]. For our approach, increasing P1 helps, which is anticipated by our theoretical bound. For the best-of approach, larger pl also helps, but exhibits population-leve optimum (p) 0.24), which is easy to 'overshoot'. Note that when prediction accuracy is low (k 0), experimentation is essential: P1 is not sufficiently large, then performance can sharply deteriorate.
DISCUSSION
Our paper studies the novel problem of learning optimal breaking policies for recommendation. We posit a tight connection between long-term engagement and user well-being, and argue that optimizing the former requires careful consideration of the latter. Viewing cognitive capacity as a limited but conservable resource, we propose Voltera dynamics as behavioral model that enables effective, responsible, and sustainable optimization of recommendation ecosystems. As any policy task that involves humans, care must be taken regarding potential risks. While our experiments show that optimizing engagement also improves well-being, this need not always be the case; in fact, merely measuring well-being in reality is challenging. Breaks, as interventions, are presumably 'safe', in the sense that at worst they may lead to suboptimal performance (for system) or satisfaction (for users). But breaks can also used nefariously, e.g., by enabling temporally-vare rewards (Fyal, 2019). As such, breaking policies should be deployed accountably and transparently.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Roy F Baumeister, Ellen Bratslavsky, Mark Muraven, and Dianne M Tice. Ego depletion: is the active self a limited resource. Journal of personality and social psychology. 74(5), 1998.
Bruce Bergum and Donald Lehr. Vigilance performance as function of interpolated rest. Journal of Applied Psychology, 46(6):425, 1962.
Junyu Cao, Wei Sun, Zuo-Jun (Max) Shen, and Markus Ettl. Fatigue-aware bandits for dependent click models. In AAAI, pp. 3341-3348, 2020.
Allison JB Chaney, Brandon M Stewart, and Barbara E Engelhardt. How algorithmic confounding in recommendation systems increases homogeneity and decreases utility. In Proceedings of the 12th ACM conference on recommender systems, pp. 224-232, 2018.
Donghui Chen and Robert J Plemmons. Nonnegativity constraints in numerical analysis. In The hirth of numerical analysis, pp. 109-139. World Scientific, 2010.
Josh Constine. Instagram says "you're 'r all caught up' in first time- well-sell-spe feature. techcrunch, 2018. URL https://tcchrrucch.om/ com/ 20//////11/////// l/scroll resposllespoossbllyy [Online; accessed 23- September-2022).
Mihaela Curmei, Andreas A. Haupt, Benjamin Recht, and Dylan Hadfield-Menell. Towards saychologically-groubkee dynamic preference models, In Proceedings of the 16th ACM Confer- ence on Recommender Systems, RecSys '22, pp. 35-48, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 450392788. doi: 10. (0.11455353222.35555 33.477778.
Shai Danziger, Jonathan Levav, and Liora Avnaim-Pesso. Extraneous factors in judicial decisions. Proceedings of the National Academy of Sciences, 108((7):689-668) 2011. doi: 10.1073/
pnas. 101 8033108.
Sarah Dean and Jamie Morgenstern. Preference dynamics under personalized recommendations. In Proceedings of the 23rd ACM Conference Economics and Computation, EC '22, pp. 795-816, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 97814503915044 doi: 10.1145/3490486.35533 3538346.
Xiang Ding, Jing Xu, Guanling Chen, and Chenren Xu. Beyond smartphone overuse: identifying addictive mobile apps. In Proceedings of the 2016 CHI Conference Extended Abstracts Human Factors Computing Systems, pp. 2821 2828, 2016.
Rachit Dubey, Thomas L. Griffiths, and Peter Dayan. The pursuit of happiness: A reinforcement learning perspective habituation and comparisons. PLoS Comput. Biol., 18(8), 2022. doi: 10. 1/371/journal.pch 1010316.
Jacob Duncan, Teresa Aubele-Futch, and Monica McGrath. A fast-slow dynamical system model of addiction: Predicting relapse frequency. SIAM Journal on Applied Dynamical Systems, 18(2): 881-903, 2019.
Nir Eyal. Hooked how build habit-forming products /Nir Eyal with Ryan Hoover. Portfo- lio/Penguin, New York, updated edition edition, 2019. ISBN 978159.
J Doyne Farmer. Market force, ecology and evolution. Industrial and Corporate Change, 11(5): 895-953. 2002.
Simona Gilboa, Arie Shirom, Yitzhak Fried, and Cary Cooper. A meta-analysis of work demand stressors and job performance: examining main and moderating effects. Personnel psychology, 61(2):227-771, 2008
Nico Gorbach, Stefan Bauer, and Joachim M Buhmann. Scalable variational inference for dynam- ical systems. Advances in neural information processing systems, 30, 2017.
Sunil Gupta, Dominique Hanssens, Bruce Hardie, Wiliam Kahn, V Kumar, Nathaniel Lin, Nalini Ravishanker, and Sriram. Modeling customer lifetime value. Journal of service research, 9(2): 139-155, 2006.
10

---

Under review as a conference paper at ICLR 2023
F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis, 5(4):1-19, 2015.
William S Helton and Paul N Russell, Rest is still best: The role of the qualitative and quantitative load of interruptions on vigilance. Human factors, 19(1)911-10,, 2017.
Robert A Hennfng, Steven L Sauter, Gavriel Salvendy, and Edward Krieg Jr. Microbreak length, performance, and stress in a data entry task. Ergonomics, 32(7):855-864, 1989.
Josef Hofbauer, Karl Sigmund, al. Evolutionary games and population dynamics. Cambridge university press, 1998.
Jiri Hron, Karl Krauth, Michael Jordan, Niki Kilbertus, and Sarah Dean. Modeling content creator incentives on algorithm-curated platforms. arXiv preprint arXiv:2206. 22022206.111222 13/02, 2022.
Nicolas Hug. Surprise: A python library for recommender systems. Journal of Open Source Soft- ware, 5(52):2174, 2020.
Ray Jiang, Silvia Chiappa, Tor Lattimore, András György, and Pushmeet Kohli. Degenerate feed- back loops in recommender systems. In Proceedings of the 2019 AAAI/ACM Conference on Al, Ethics, and Society, pp. 383-390, 2019.
Daniel Kahneman. Attention and effort, volume 1063. Citeseer, 1973.
Dimitris Kalimeris, Smriti Bhagat, Shankar Kalyanaraman, and Udi Weinsberg. Preference am- plification in recommender systems. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pp. 805-815, 2021.
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. The challenge of understand- ing what users want: Inconsistent preferences and engagement optimization. arXiv preprint 2277::2200..1776, 2022.
Robert Kleinberg and Nicole Immorlica. Recharging bandits. In 2018 IEEE 59th Annual Symposium Foundations of Computer Science (FOCS), pp. 309-319. IEEE, 2018.
Karl Krauth, Sarah Dean, Alex Zhao, Wenshuo Guo, Mihaela Curmei, Benjamin Recht, and Michael Jordan. Do offiine metrics predict online performance in recommender systems? arXiv preprint ar Xiv::220 1.00931,, 2020.
Liu Leqi, Fatma Kiline Karzan, Zachary Lipton, and Alan Montgomery. Rebounding bandits for modeling satiation effects. Advances in Neural Information Processing Systems, 34:4003-4014, 2021.
Alfred J. Lotka. Contribution to the theory of periodic reactions. The Journal of Physical Chemistry, 14(3):271-274, 1910. doi: 10. 10.1021/j1501 1a004.
David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: improving fairness and accu- racy by learning defer. Advances Neural Information Processing Systems, 31, 2018.
Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. Feedback loop and bias amplification in recommender systems. Proceedings of the 29th ACM international conference on information & knowledge management, pp. 2145-22 2148, 2020.
Alberto Monge Roffarello and Luigi De Russis. The race towards digital wellbeing: Issues and opportunities. Proceedings of the 2019 CHIl conference on human factor in computing systems, pp. 1-14, 2019.
Christian Montag, Zhiying Zhao, Comelia Sindermann, Lei Xu, Meina Fu, Jialin Li, Xiaoxiao Zheng, Keshuang Li, Keith M Kendrick, Jing Dai, al. Internet communication disorder and the structure the human brain: Initial insights on wechat addiction. Scientific reports, 8(1):
1-10, 2018.
11

---

Under review as a conference paper at ICLR 2023
O' O'Dhaniel A Mullette-Gillie Ruth LF Leong, and Yoanna A Kurnianingsih. Cognitive fatigue destabilizes economic decision making preferences and strategies. PloS one, 10(7):c0130022, 2015.
Mark Muraven and Roy F Baumeister. Self-regulation and depletion of limited resources: Does self-control resemble muscle? Psychological bulletin, 126(2):247, 2000.
Sarah Perez. Apple unveils a new set of 'digital wellness' features for better managing screen time. TechCrunch, June, 4, 2018. URL https://techeruncc.comm http://rrnnnnccooooo/oooeooo http://00//006//////--p///-e a new-set- -atprr/mmaaaanngg-ee----iiieeee [Online; accessed 23-September-
Seegital-wellless-ffieeeeee 2022].
Jennifer Rauch. Slow media: Why slow is satisfying, sustainable, and smart. Oxford University Press, 2018.
Stephen E Robertson. The probability ranking principle in ir. Journal of documentation, 1977.
Hayden A Ross, Paul N Russell, and William S Helton. Effects of breaks and goal switches on the vigilance decrement. Experimental brain research, 232(6): 1737, 2014.
Tom Ryder, Andrew Golightly, Stephen McGough, and Dennis Prangle. Black-box variational inference for stochastic differential equations. In International Conference on Machine Learning, pp. 4423-4432. PMLR, 2018.
Larry Samuelson. Evolutionary games and equilibrium selection, volume MIT press, 1998.
Francesco Sanna Passino, Lucas Maystre, Dmitrii Moor, Ashton Anderson, and Mounia Lalmas. Where to next? dynamic model of user preferences. In Proceedings of the Web Conference 2021, WWW '21, pp. 3210-3220, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383127. doi: 10.1 1..114//44238..3444 3450022..
Sven Schmit and Carlos Riquelme. Human interaction with recommendation systems. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 862-88 870. PMLR, 2018.
Maarten P. Scholl, Anisoara Calinescu, and Doyne Farmer. How market ecology explains market malfunction. Proceedings of the National Academy of Sciences, 118(26):e205557411 2021. doi: 10. 10.1073/pnas.201557)
Hans Henrik Sievertsen, Francesca Gino, and Marco Piovesan. Cognitive fatigue influences stu- dents' performance on standardized tests. Proceedings of the National Academy of Sciences, 113 (10):2621-2624, 2016.
Yasuhiro Takeuchi. Global dynamical properties of Lotka- Volterra systems. World Scientific, 1996.
Dustin B Thoman, Jessi L Smith, and Paul J Silvia. The resource replenishment function of interest. Social Psychological and Personality Science, 2(6):592-553 -999,99999900 2011.
Emmanuel Trélat and Enrique Zuazua. The tumpike property in fnite-dimensional nonlinear opti- mal control. Journal of Differential Equations, (58(1):81-114, 2015.
Mengting Wan and Julian McAuley. Item recommendation monotonic behavior chains. In Pro- ceedings of the 12th ACM conference on recommender systems, pp. 86-94, 2018.
Mengting Wan, Rishabh Misra, Ndapa Nakashole, and Julian McAuley, Fine-grained spoiler detec- tion from large-scale review corpora. arXiv preprint arXiv: 13010....411.... 2019.
Jyun-Cheng Wang and Juo-Ping Lin. Are personalization systems really personal?-effect of con- formity in reducing information. In 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings the, pp. 10 pp. IEEE, 2003.
Romain Warlop, Alessandro Lazaric, and Jérémic Mary. Fighting boredom in recommender systems with linear reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Jörgen W Weibull. Evolutionary game theory. MIT press, 1997.
12

---

Under review as a conference paper at ICLR 2023
A DEFERRED PROOFS
In this section, we formalize the model presented in section 3, and prove the claims presented in section 4. The section ends with formal proof of Theorem 1.
A.1 PROPERTIES OF LOTKA- VOLTERRA SYSTEMS
Definition A.1 (Static policy equilibrium). Let A(L). q(t) denote a Lokta- Volterra model character- ized by parameters 0 (a, 7,6) R4, as defined in Equasioo Let p E [0, 1), and denote by Tp the static policy corresponding to For A(0), (()) > 0, the static equilibrium of the system is defined as:
(x; 0) lim X(t)
q (p: 0) lim g(t)
(p; 0) when 0 is clear from the context. We denote X u) (p: when a user u characterized by parameters 0₂ given and clear from the context.
We denote (p)
Proposition A.1 (Global stability). X (p; () exists and uniquely defined for all 0 IR4, p [0, and for all initial conditions X(0), ((0) > O.
Proof. See (Takeuchi, 1996, Section 3.2).
Lemma A.1 (Equilibrium of LV behavioral model. Formal proof of Lemma 1). Assume Lokta- Volterra model characterized by = (a, 7,76, 6) E TR4, and let E [0, denote the proportion of interactions in which a forced break is served. The static equilibrium of the model under static policy n is given by:
p 0, otherwise
(p)
I-P
0
(p)
B
T-p
De
otherwise
Proof. The LV dynamical system is given by Equation 3:
yl dt aλ + 3(1 - p)>q
dq
rg(1 - 4) - 5(1 p)>a
when p E 0. we equate de 0, and obtain the result. The solution is guaranteed to be valid, as both X (p) 0 and q" (p) [0. 1].
dA
d Conversely, when & 0. / there exists e > 0 such that āz log A < - < 0 for all À 0, E [0, 1]. From this we obtain that log A(t) tends towards -oo, and therefore A(t) tends towards 0, and (p) O as required. When A(t) is close to zero, the interaction terms vanish in the dg equation, and (t) grows logistically towards
Proposition A.2 (Equilibrium bounds). For Voota-Vottera model, the static equilibrium X x²)p) is bounded by:
B8
S (p) v
4a8
Proof. Denote x
I-p 1. From Lemma A.1, for x E
[1, [
the equilibrium consumption X (x))
given by:
X (x)) = 2
a
-
3
13

---

Under review as a conference paper at ICLR 2023
and is zero otherwise. The equilibrium a quadratic function æ with roots x {o, = and
therefore attains its maximum at x Plugging back the maximizing into X we obtain the upper bound. Lower bound attained as the equilibrium in Lemma A.1 is clipped by from below.
Lemma A.2 (Optimal static policy, Formal proof of Lemma 2). The optimal static policy for Lotka- Volterra system given by:
1-29
2g
K
Popt
And the optimal equilibrium engagement rate is given by:
fr
1
1ad
2
Xopt
18
1
Proof. Denote x I-p' From Proposition A.2, the global maximum of X (a:) is attained at $ Consider two cases: When S 2. we obtain that Zopt 2a 2 1, and therefore Popt = 1 - 1 E [0, 1]. From this obtain that in this the global maximum is attained on the simplex, and given by the formula from Proposition A.2. Conversely, when > 7. we obtain = 1 x < 0, and therefore Topt translates to negative value of As X (p) is uni-modal, the optimal policy restricted to the simplex [0, 1] in this case is attained on the closest boundary point p = 0.
Figure 2 provides graphical intuition for this proof (left and center subplots).
Proposition A.3 (Inference of a/B from two-treatment equilibrium data. Formal proof of Proposi-
Lokta-Volterra tion 1). Let (()) be Lokta- Volterra model, lel P1,P2 [0, 1]. Denote by X (p1), X" (p2) the
static equilibrium rates corresponding to static policies Tp Tpp and assume X (p1). A (p2) > 0. The parameter ratio given by the following formula:
P2- (p2) -1- (1-)))A((1))
B
T-pi
I-pa
Proof. From Lemma A. the equilibrium consumption X is given by:
x²))) = 7
1-
1
3 31-p
7 81
a7
1
2
p
B
1 1-p)
When X (px) is observed for different policies Pi...., E regression problem for the parameters and e which can be solved e.g using Non-Negative Least Squares.
|0,
/ we obtain a polynomial
f 8
When = 2, obtain system of two linear equations. Apply Cramer's 's rule obtain:
X (pa)
X (p1)
(1 - p2) (x2) - (1 - 21
p1
(1 P2)
(11)
8
(1 (1-P1) Pi [1 -Pa
(1-P1) (11-pa)
P2 P1
x (p2) (p1)
a7
(1 (1-P1) (1 (1-p2)
pI) p2) (1- p2)2 (p2) (1 (p1)
38
(1-p:) (1-p2)² (1-p:) (1-p2))
P2 P1
(12)
14

---

Under review as a conference paper at ICLR 2023
And therefore is given by:
(pa) (p(p)
(1 p1)(1 P2) (1 - p2) (p)
a [1-P1] (I-Pz)
3
(pa)
(P1)
(1- -21-
(1 (1-P1)² (1-Ps)²
A.2 MODEL FITTING FROM ENGAGEMENT PREDICTIONS
Notations. In this section only, the common notation q p denote complementary probabilities.
Definition A.2 (Empirical value of a/8). For single-channel experiments with forced-break proba- bilities pr P2. denote X; X2 (p.). fo (u), q: 1 1-pt. The empirical value of the 3 parameter given by the following formula:
41 (41 f 4212)
3 of f 22
Proposition AA (a/3 estimation error from prediction errors). Given single-channel Lokta- Volterra system with parameter 39 1. Let P1, E [..g]. denote X (p,))) R4. and
let Ji X + & be the predicted engagement rales corresponding to p1,22 When lea |62l S e s
ppl- the estimation error bounded by:
35
B
P1
P2
ory
Proof. denote = 1 - -ps. The value of given by Proposition A.3:
a 91 (9 9225)
B
oi A
And the estimator for 3 is obtained by replacing the true value with their predictions:
à 9192 (91fi 92222
8
qi 42.22
9192 (9) (2 + =1) - (2( E2))
iii + - (2g +
The estimation error is given by:
à
0i (11 (22 x1
B
{9 xi
(
25 (afe1 9322)
)
91 42
E2 E
22
(gfer 22
=0
=
(iv)
We now proceed to bound each factor:
For (i), the term (q192)² 2 bounded by since q1, 42 E [0, 1).
For (ii), the term
equal (3) by Eq. (11).
For (iii), from Proposition A.2 we obtain the bound < X <
4nd and therefore the term
9y
| - E Ag1 is bounded by
fiy 100
By 2
8.
15

---

Under review as a conference paper at ICLR 2023
For (iv), the term
is equal to:
0f - (0.81 0262)
(iv)
?7 12 (9f2) - 02
as 1/ (4fe1 0282)
P1 - P2l
pr - P2
1
G 02 9i 02
[P1 P2
P1 P2
P1 P2
g 42
P1 P2
Note that gee 22
2r
When e is small enough, and specifically when the bound
P1--aal
P2l holds, obtain:
52
S 2 8
S
P1
P2
and therefore:
2 6
(iv) S
PP1 P2l
Aggregating results (i)-(iv) above, we obtain the overall bound:
à
)
91
92
E2
E
1
B
2
23 (9fE1
$1
2 p1 P21
8 35
P1 Pal
acy
Proposition A.5 (Cost of a/9 estimation error). Let 3 be the engagement ratio parameter of one-channel Voltera system, and let ( be an estimate of these parameters. Let Xopt be the engagement rate of the optimal static policy, and denote (x) X* (f(z)). When
B
s
min 27 1} The price of estimation error is bounded by:
-2
topt - 1"
(2) min
Q
3
3
Proof. Denote r r= -g, and assume without loss of generality that = and r S 1. The optimal equilibrium engagement rate is given by:
r (0, ! Aopt -- re (
The chosen policy f(x)) is given by:
11 2x 10
p(x)
0
otherwise
16

---

Under review as a conference paper at ICLR 2023
Cost estimation error different values of 5, corresponding upper bounds
0.30
0.8
0.50
0.4
0.0
-0.6
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
Ag (9)
Figure 4: Graphical illustration of Proposition A.5. Cost of estimation error for different values of g. and their corresponding upper bounds given by the claim.
Assume without loss of generality that E [0, 2) as values of outside the interval can be clipped to its edges without affecting the result. The equilibrium engagement rate of the selected policy is given by:
E 0, x
(x) X (f(x))
(1
2
Denote A(x) Xopt (z)). We obtain:
E (0,
x E [0, 5 (r
7
A 2 Xipt (²)x)=
(x-r)²
422r
r E (0, 11
x E
2
(1 - r)
EE (),
x E 0.
r)
2x (1
2.)
rE 2
2 21
Observe that 1 2 1 - for all r E (0, 1]. and therefore we obtain for all d,
är
A (x)) S
(13)
4r
From the convexity of Axx) in the region around x we obtain:
A -
2 x rl
(14)
Finally, combining the two bounds yields the final result. A geometric interpretation of this claim is illustrated in Figure 4.
A.3 OPTIMAL STATIONARY POLICY FROM ENGAGEMENT PREDICTIONS
Definition A.3 (Expected observable rate). Let u E U, E [0, 11. and T > 0. Let p E [0, 11, denote the corresponding static policy by Tp. The expected observable rate >u(p; T) is defined as:
Nu(y; u) = =x TPP
TPP (u; (u;T)]
where expectation taken over the stochastic decisions of Tt.. Definition A.4 (Lokta-Volterra approximation of TPP). Let u E U, and T > 0. Denote by p" the maximizer of expected observable rate:
arg max (p; pE[0.1]
The LV approximation of TPP(u; T) is defined as:
0- arg min max
() - X² (p: pE(o.1]
0
such that arg max, X" (p; p". The corresponding approximation error is defined as:
CLV. = max [Ta(z u) - X (p: 0%)
pE[o.1]
17

---

Under review as a conference paper at ICLR 2023
Notations. When u is clear from the context, we denote 0" f. ELV ELV,u. We a", to refer to the corresponding parts of the Lokta- Volterra parameters vector 6*
We now ready to state and prove the main theorem for this section:
Theorem A.1 (Regret bound for learned static policy. Formal version of Theorem 1). Let p1 P2 E [0, 1] denote static forced-bree bbeek policies, and denote by U the set of users, and assume they main engaged under the stationary policies π (p))) and π (p2). Assume (u(p; T) ~ TPP mpo (w: T),
Ju: and let μr (maxuel b..) (max_ee (nd ) maXueu
Let Sp2 :u + R4 be functions predicting 11,, (PI:T),, 1Ss, (p2:T)), respectively. Denote the learned policy by P. and the optimal policy by p".
If (i) the expected RMSE of fpr- is bounded by Epred. (ii) the average absolute deviation of + (TPP(u;;)) is bounded by Edev, and (iii) the expected LV approximation error of the system is bounded by ELV, then the learned policy P has bounded regret:
Eu. (p": 115 (o:T))l
1/PP
(Epred + Edev + ELV)
P1
P2l
where expectation is taken over stochastic choices of policies, and fyPp g(u, v) E poly(u,
Proof. By assumption (i), the functions fps fpz have bounded expected RMSE:
En [J (u) ISs (p:: T))) 21 S prod
(15)
Applying Jensen's inequality with the convex function co(x) = x yields:
E.. (u) Is. (p:: TIII)) E,, (Jn, (u) + Iw. (pe (p&T))) T))"
Combining with Eq. (15) and taking the square root, obtain upper bound the expected absolute error:
(w) +s. (p: T)| pred
(16)
Let A = fp. (u) - X² (px) apply the triangle inequality to obtain:
A = 1p, (u) (p:)
< fp. (u) 11s. (u: T)| + I1s. (u: T) - M(p u) + (1pp; X (xx)
Denote 5 = Epred + Edev + Lyy Applying the triangle inequality and using the bounds in Eq. (16)
together with assumptions (ii). (iii,, we obtain:
E s E Ppx (u) - Sls (u; T)]
B [| |15, (u: T) X(paw)] + E. [xppr u) - X (p::
Sepred + Edev EIN = E
(17)
Denote 0; = (a, B, 7, 7). The empirical value
3 of ($) is given by Definition A.2. Denote the
: -(3)
estimation error by Ay =
By Proposition A.4, the following pointwise upper bound on Ag applies when A, 5 (pi-pal
f
38
(18)
P1 P2l ar
Plugging in the bound the expected value of A, into Eq. (18), we obtain in expectation:
1P1
p2
A
36
7 |pi
P2]
E
A
A
E
A
4
P2
ary
4
85
98
(19)
P1
P2
ary
18

---

Under review as a conference paper at ICLR 2023
Next, we apply Proposition A.5. Denote A = ->"(p))-A"(") and define the following probability event:
=
A
7 5
P1
p2
and
2v
Note that the bound in Proposition A.5 is represented as a minimum between two functions, one linear in and one constant. To leverage this property, apply the law of total expectation:
E A = E., A
A]P[A] E., A P]
(20)
Under A. the first term in Eq. (20) can be bounded by the linear term in Proposition A.5. Taking P[A] & 1 and combining with equation Eq. (18):
Eu, |A,- ||][[]] & Eu, A A
I Bu,
A
|
20
A
135
E
|A
2a
P1
P2
ary
33
Es
(21)
2/p1
P2
The expectation factor in the second term of Eq. (20) can be bounded by the constant term in Propo- sition A.5:
By V
Eu
A
A| - 4
max
(22)
Decompose the probability factor
A using the law of total probability:
1p1
P2
7 1p1 - P2
7 1p
p2
P A
=
P
A
+
P
A <
P
A
5
4
2v
4
8
4
PI
p2
P1 P2
P
A
>
+ P
>
A f
<
5
2v
4
Apply Markov's inequality
P|[X|
2 a)
v
X[|]]]
on the probabilities to obtain:
7 p1
P2l
(
1p1
p2
P
A
Eu. p
8
6 4
4
El
max
(23)
(17)
1p1 - P2l
a
1
7 P1
P2l
Y
PP1
P2
P
A
<
E
A
S
2v
8
4
38
max
by Eq. (19)
P1 - P2
Ef
max
(24)
PP
P2
Plugging back equations Eq. (21), Eq. (22).Eq. (23),Eq. (24) into equation Eq. (20), we obtain bounds for each term:
Bu 1x-1 Ex A[P[A] + E, A A
P
(25)
(21)
Eq. (22)
by Bq. (23).Eq (24)
we obtain:
33
222
Egg [Ax]) 5
Es
+
v +
11
=
P1 P2
2
4
19

---

Under review as a conference paper at ICLR 2023
To obtain the regret bound on the empirical rates, we apply assumptions (ii), (iii) again to bound the expected difference between A (p) and 1 SS (p: and apply the triangle inequality:
E uAy
| (p' :T) Is. (D:T)Il & Ex - 2( + +Ev)
Note that pp--2l
z since all the users are assumed remain engaged in the S as P1.P2 E [0, 1]. Therefore, the function TrPP = g(u, v) =
long term, and Pp1 P2
2 +
17 + 2y
satisfies:
1
Eu, [|s, +15, (A:T) 2
4TPP (Epred + Edev + SLV)
P1 P2
B EXPERIMENTAL DETAILS
B.I DATA
MovieLens-IM We base our main experimental environment the MovieLens- 1M dataset, which is standard benchmark dataset used widely in recommendation system research (Harper Konstan, 2015). The dataset includes 1,000,209 ratings provided by 6,040 users and for 3,706 movies. Ratings are in the range (1....,5) 5), and all users in the dataset have at least 20 reported ratings. The dataset is publicly available at: https://grouplens. https://roupllnn.org/aaaaaasstt org/ddttaeet/mooviilnnn// lm/.
Data partitioning. To learn latent user and item features, 30% of all ratings were drawn at random. Stratified sampling was applied to ensure that all users and items were covered, and so that each users have roughly the same proportion of ratings used for this step. These ratings learning CF model, and were discarded afterwards. The remaining 70% data points were used for training and testing. For these, first randomly sampled 1,000 users to form the test set. Then, the remaining users were partitioned into the main train set S, which included 70% (Rx35528) of these users, and the experimental treatment sets 861,, each including 10% (=504) users for N 3. This procedure repeated 10 times with different random seeds. We report average results, together with 95% t-distribution confidence intervals representing variation between runs.
only used only for
B.2 IMPLEMENTATION DETAILS
Hardware: All experiments were single laptop, with 16GB of RAM, MI Pro processor, and with no GPU support.
A single run consisting the entire pipeline (data loading and partitioning, collaborative filtering, training classifiers, simulating dynamics, learning policies, measuring and comparing performance) takes roughly 23 minutes. The main bottleneck the discrete LV simulation, taking roughly 70% of runtime to compute, mostly due to bookkeeping necessary for the non-stationary baselines. Simulation code was optimized using the NUMBA jit compiler, which improves run- time.
Runtime:
Optimization packages:
Collaborative filtering (CF): We use the SURPRISE package (Hug, 2020), which includes an implementation of the SVD algorithm for CF. All parameters were set to default values.
Regression: We the SCIKIT-I.EAR implementation of linear regression for predicting long-term engagement from user features (i.e the prediction models fz(u) in Eq. (8)). All parameters were set default values.
Non-Negative Least Squares (NNLS): We use the SCIPY. implementation of NNLS. The algorithm was used with its default parameters.
Code: Code for reproducing all of our figures and experiments is available in the following repos- itory: https://github.c hwtp://oooo//mlmoo/mo222030-
20

---

Under review as a conference paper at ICLR 2023
B.3 OTHER BASELINES
Safety: In each step of the TPP simulation, look step back, and calculate the empirical rate X; = t-4; x. If this rate exceeds the threshold b T, the policy enters a 'cool-down' policy state, serving only forced breaks until the next time period. In our experiments, we used thresholds {14, 16). & 10 book-behind steps, and defined the cool-down period as 0.5 time units.
Oracle: To estimate the effect of perfect predictions, implement oracle predictor f (u) which has access to the latent user parameters. For a given u and for each p. the predictor outputs the infinite-horizon LV equilibrium for u, namely fp foracle(u) (u) (p; (x). We define
0, = (au, Bu. Tu: 8u), where Ta: On are the unobserved parameters for the given user, and B. is the expected value of Sur induced by the distribution over recommended items I induced by the recommendation policy tb. We view 6, as useful proxy for the otherwise unattainable 8...
B.4 HYPERPARAMETERS
Collaborative filtering: We used 8 latent factors and enabled bias terms, which ensured per- formance is close to the benchmark of RMSE 0.873 reported in the SURPRISE documentation. We used the vanilla SVD solver, with all hyperparameters set to their default values.
Recommendation policy: Softmax temperature was set to 0.5.
Prediction: We trained regressors (u) on input feature vectors consisting of three components:
u = (ö,, bu, Pu) E d+2
(26)
The three components are: (i) SVD latent user factors in E RR,, (ii) SVD user bias term by E R. (iii) additional feature consisting of the average predicted ratings for unseen items f. weighted by recommendation probability, which we found to slightly improve predictive performance:
Pu =
Σ
Fus softmax,
(77)
=Eholdout(a)
Where holdout(u) is the set of unseen items corresponding to user u, faz [1, 5] are the pre- dicted ratings, and P. [1,5] hoodoutt(]] is the vector of all predicted ratings used for softmax recommendation as described in section 5. We chose to focus on linear models since the treat- ment datasets are relatively small (each [S(s)] ~ 500), and since other model classes (including boosted trees and MLPs) did not perform significantly better.
Discrete TPP: Interaction sequences for each user were generated according to an LV dis- cretization scheme TPP (w: T), described in detail the next section. Latent sates were initialized randomly with relative uniform noise around the theoretical LV equilibrium point (Ap, qn) = (1 + Ex)A*, (1 +1+ E,)4"), where Ex ~ Uniform( (-011, 0.1). Latent states were up- dated each B = 10 recommendations to stabilize noise (see Figure 5). When is recommended to u at time t, latent states and At are set according to (t), which depends on ratings Tua (true or mixed with predictions u via x). Specifically, we use (t) r / {0.2, 0.8, 1.8, 3.2, 5), which is convex, to accentuate the role low ratings since they are underrepresented take the effective (t) be the average
the data. the B items recommended in that step. We set a = 1.3, and chose 7 0.2, = 0.01 (which together determine scale) so that typical values for engagement rate 7 [S.] are on the order of A 10 for the chosen T = 100.
For B 2
B.5 DISCRETE TPP FOR LOKTA- VOLTTAR SIMULATION
The Temporal Point Process (TPP) on a discretization of the LV system described in Eq. (2), using the forward Euler method with variable step sizes. We denote this process by TPpLV (w; T"), and present the sampling procedure in Algorithm
for simulating user interaction sequences S. is based
Each user associated with discrete latent states A gis and parameters aur Yus Su. Initial states An- qu are set randomly. At each step, and in time t,, the system recommends x, z(tt), which triggers updates in latent states, and determines the next time of interaction 4,41. As noted, these update depend item-specific parameters Bu,a;-
21

---

Under review as a conference paper at ICLR 2023
Discrete simulation (B 1)
Discrete simulation (B 10)
Continuous ODE
Latont consumption dive 2015
610
40-
00
Cumdlative rate
Throrttical
Figure 5: Example discrete sequence S. TPPLV (u;T), T), compared to continuous LV dynam- ics. Our TPP produces discrete sequences that are qualitatively different from their continuous- time analogs (blue lines), Nonetheless, it captures the general properties of our proposed behavioral model: note how cumulative averaging behavior (orange dashes) exhibits 'habit formation', which equilibrium approach targets (blue dots). For the same initial conditions X(0), ((), the figure shows how varying the number of recommended items per step (B) 'smooths' the discrete behavior (left: B 1, center: B 10). For fixed Bt(() Bu. when B - oo, and when At TPP sequences approach a continuous LV trajectory; in general, and particularly when Bu(() varies by step and per recommended items this the case.
Under stationary policy =p). the system recommends an item with probability (1 (1-p), and suggests break with probability p. The simulator considers B recommendation opportunities each step. For each & E {1...., B). denote by T E 10,1} the break indicator, equal to when a break recommended the k-th slot the batch. Denote by &b the item recommended by the underlying policy 4b, and by B(x) = / the corresponding LV hyperparameter as defined above.
Algorithm 1: Sample from TPp (w; T))
Input: Break probability p E [0, 1]
Stationary content recommendation probability &b
Voltera parameters 6, = (0,
Time horizon > 0
Output: Interaction sequence S, TPPLY R(njow (w;
Initialize i= 0, to S. ():
while 4 < <d
foreach E (1...., .......P. B} do
Ia Bernoulli(1 p):
IR ~ 1h;
3x B(ruxa);
end
At;
1,1;
\ (-a+ Ee_l laBn q: A,Ar;
A++1
a+1 - qi + (7(1 q:)
Σ Σb 126 X1) A.
t+ - t; + -
S, - Sy U [(li, ((...__)))))
it i+1;
end
22

---

Under review as a conference paper at ICLR 2023
C ADDITIONAL EMPIRICAL EVALUATION
In this section, we provide additional empirical results as further indicators potential applicability:
In subsection C.I, we repeat the experiments presented in section 5 using different dataset, showing that qualitative results remain consistent across recommendation modalities.
In subsection C.2, demonstrate how to leverage additional problem structure commonly avail- able in practice, and develop adaptive breaking policies which improve over time.
Finally, in subsection C.3 we present stateless consumption model which is unrelated to Lotka- Volterra and does not promote breaks. Evaluation of the LV policy optimization method under this model shows that the optimization procedure does not degrade performance when breaks are not beneficial, thus extending Corollary 2 beyond the realizable case.
C.1 ADDITIONAL DATASET
The Goodreads dataset. The Goodreads book recommendations data set is a common bench- mark dataset used in recommendation systems research (Wan & McAuley, 2018; Wan al., 2019). Ratings are in the range {1..... .......5,, 5). and the dataset is filtered to only include users with at least 20 reported ratings. We the official comic-books genre subset of the dataset, which includes 3,679,076 ratings provided by 41,932 users and for 87 87,565 books after pre-processing. The dataset publicly available at: https://sstes .owgle..oooo.ooo...o....o..... google.com/eng ucsd. ccu//ucsdbookgraph. Pre-processing code is available in our code repository: https://github.co www.//coommmoo/eee.eomemm -ttp////mmllll222//m....
Evaluation. For evaluation, we follow the same procedure described in Appendix B, maintaining identical data processing procedures, constants, and hyperparameters. This procedure was repeated 10 times with different random seeds. We report average results, together with 95% t-distribution confidence intervals.
Results. Figure shows results for the Goodreads experiment, in the same format as Figure 3. Results exhibit performance and trends that are qualitatively similar to the MovieLens experiment section 5. The LV policy optimization method achieved better performance on this dataset (right pane): The performance of the LV is closer to oracle e (-0.526% in Goodreads compared to -0.918% in MLIM), and the gap from the best-of method is slightly larger (+5.77% in Goodreads compared to +2.19% in MLIM). Varying user types (center pane) shows less variation across values of x, indicating that predictors achieve satisfactory performance even when the x is low and prediction harder (see subsection 5. Varying treatments (right pane) also coincides with the observations: LV policy optimization on the Goodreads dataset exhibits less performance degradation as Pi + O, suggesting that less data may be sufficient for optimization. We attribute these results to the larger size of the dataset (approximately 3.6M interactions, compared to IM interactions in MLIM), and the possibility that stronger structure may exist in this recommendation modality compared to general movie recommendation.
Engagement for different policies (R 0.5)
Engagement per LV breaking schedule
Engagement for varying (N
Improvement (vs. dobait)
12%
Imgrossment (ox. dedadt)
23%
LV decision rule
improvement (ve. detaiit)
10%
20%
8%
15%
8%
LV,
5%
10%
5%
LV. x 0.5
3%
5%
7%
best of, /
0%
0%
0.5
defaidt
0%
bend-oof of,
defkoft
bestel
Gradle
0.1
0.2
0.3
0.0 Assigred treabment (xx)
11.2
0.4
Learned LV policy (6.)
Figure 6: Results on the Goodreads comic-books dataset (compare to Figure 3). (Left) Perfor- mance gain of different approaches (relative to default policy). (Center) Performance of LV by user group, partitioned by learned policies Åu. (Right) Sensitivity an increasingly aggressive ex-
perimental (N = 2, 0).
23

---

Under review as a conference paper at ICLR 2023
C.2 LEVERAGING ADDITIONAL STRUCTURE USING ADAPTIVE POLICIES
One benefit of our approach in section 4 is that it requires only predictions of long-term engage- ment. In reality, however, other sources of information may also be available to the learner, which may be useful for improving decisions regarding when to recommend breaks. Here we consider additional information in the form of explicit user feedback, collected by the system over the course of interaction. We model users as providing to the system feedback regarding item quality, namely true ratings Fur. for some of the recommended items x they consume. For simplicity, we assume users report ratings with probability Pr E [0, I], independently for each item, where we vary Pr across experimental conditions. Intuitively, we would expect that, over time, user-reported ratings help improve our learned breaking policy; however, over-relying on few data and at and early stage my hinder performance. Here we examine when and how such feedback should be utilized.
Adaptive policy optimization. To extend our approach to incorporate ratings-based feedback, approach which can leverage sparse explicit feedback. Fixing an adaptation time To E [0, I] (a hyperparameter), the method applies the standard learned breaking policy से (Equation 9) in the time frame S To. during which it also collects and stores user-reported ratings. Then, at time f = To. the method updates the learning policy on the basis of the new ratings data, and applies this policy until the eventual time T. In particular, ratings are used to improve the estimated component P., in the user feature vector u' (described in Equation 27). In this way, the method uses the existing engagement predictors (u) without requiring any retraining.
we propose an adaptive heplore-then-explon
A formal description of this method is provided in Algorithm 2. Here denote the set of rat- ings collected until time by *[o.e] E {1,.....5 5)"; the average rating at time t by P(o.e) = Mo.4 Enerpona and the set of long-term engagement predictions for feature vector u by (u) (pr, Sp, (u))) (as in subsection 4.1).
Algorithm 2: Adaptive policy optimization using sparse rating signals
Input: Initial break probability p E [0.
Time horizon > 0
Adaptation time To E [0, 7)
User feature vector u = (8,b,
1. Collect ratings data [[u,Tb] with breaking policy (p) until time To;
Construct updated feature vector u' (ū, b, ())) using the average rating Plo.tal :
3. Compute updated long-term engagement predictions f(r');
4. Use the LV policy optimization method to obtain updated policy p' = p" (f(")))
5. Use breaking policy (p') for the remaining time t E [],
Evaluation. We evaluate the adaptive policies using the MovieLens- 1M experimental setup, as described in section and appendix B. We make use of the same linear regression predictors (p(u) from the main part of the experiment, and maintain the same time horizon T 100. We vary the ratings density parameter p. in the range [0, 1], and vary the adaptation time To across three distinct values To (0.5, 5, 503.
Results. Results are presented in Figure 7. In the results, we observe that LV-adaptive with To = represents optimal point within the hyperparameter space. For this choice, results show significant gains over the non-adaptive LV method even under moderate rating densities, and quick convergence towards the LV-true- -ratungs upper bound. In contrast, LV-adaptive with To 0.5 represents a "premature optimization" " scenario, attempting to adapt the policy without allowing the estimated average P(o.To) enough time to converge to its expected value. Similarly, To 50 represents the result of "late optimization", which has accurate estimation of Plo.T.) but not enough time to benefit from the updated policy. Finally
C.3 DISTINCT BEHAVTORAL MODEL
In this work, we propose the LV model as a behavioral hypothesis class for counterfactual prediction of long-term engagement. As such, using the L.V model within the learning-tor -brakk optimization
24

---

Under review as a conference paper at ICLR 2023
Performance LV-adaptive policy (i = 0.O.T 100)
0%
Improvement over mypplc poloy
Policy
0%
IVadapiio (% 5.0
[V-adapiive (2b 0.5)
IX-atlapios 5200)
6%
orace
ratings
2%
IV
defailt
0.9
0.2
044
0.6
088 1.0
Rating dursity 24
Figure 7: Adaptive policy evaluation for varying rating density pr [0, 1]. Solid blue line rep- resents the adaptive policy described in Algorithm 2, with adaptation time To = 5. Dashed lines represent the performance of adaptive policies under different choices of To. Cyan horizontal line represents WV-true-ratings a non-adaptive method with oracle access to the true average rating. The remaining horizontal lines represent the performance of selected non-adaptive policies, as pre- sented in Figure (left pane).
framework is a design choice, to be made at the discretion of the learner; LV model would be good choice if it fits the data better than alternative model classes, given the amount of available data. This relation is made precise in error bound (Theorem 1), which bounds the error when using the LV model for any underlying TPP (i.e., we make no assumptions about the true underlying data generation process).
Our main experiments evaluate our approach on data that is not LV dynamics, but nonetheless, bear some resemblance. To complement these results, here we we empirically evaluate our LV-based approach on data that is generated by a behavioral model that entirely distinct. In particular, we consider user model in which consumption decisions only depend on the quality of recommended items, without any dependence on internal states-i.e.. it is stateless. This conforms behavioral models which are implicitly assumed in conventional recommendation methods such as collaborative filtering.
additional experiment in which
Stateless content consumption. The data generation process formalized in Algorithm 3. As means of capturing stateless behavior, define "close-range" temporal point process TPPCR (w: T) which generates sequences of user interactions based solely on the average rating of recommended items in a batch. Here we relate rating with utility, and assume that items having higher utility (and therefore higher ratings) induce more frequent interactions: in the same way, consider break prompts as items having zero utility, and hence zero rating. At time 4. and given batch of size B with & recommended items and B k 2 breaks prompts, users acting according to the stateless behavioral will consume the next batch of content at time:
k
1
t++1 = =t
Tuzy
(28)
5)* the ratings of the items recommended at time k. and 7 > is a constant latent parameter to be learned from data. The breaking policy π decides on the number of items A recommend each step. Since Tur 2 for all user-item pairs, the time difference At, = 441 4 in Equation 28 is minimized by taking k = B. This shows that the optimal breaking policy under this behavioral model the default one, which does not prompt the user to break.
where (Fuz. T'ur....., Ture)
We evaluate the stateless behavioral model using the MovieLens- 1M experimental setup, described in section 5. We set 4 for all users, utilize linear regression for engagement prediction, and maintain all hyperparameters without change. Data processing steps are performed described in appendix B, and the chosen breaking policies evaluated.
Evaluation.
25

---

Under review as a conference paper at ICLR 2023
Algorithm 3: Sample from TPPCR Tu;
Input: Break probability p E [0, Stationary content recommendation probability vb
Scalar parameter 0n, = >0 0
Time horizon 7 >> 0
Output: Interaction sequence S, TPPER
=(p)o (u; T)
Initialize i i= 0, to = 0, S. 0:
while < ' do
foreach k E {1.... B do
I Bernoulli(1 p):
1/;
end
4++1 - 4 + + 1
1 Lk=1 INTnr
-
Su Su f(e,, (I1...., NB), (I1...., IB));:
i i f-- i 11;
end
with unbiaced prodiction
IV fit with ever-prediction
15
Concamptio 10
Concumption
Expected empirical ongagement 3,
Engagement prediction (n. (2.0))
Best fit LV equilboium curve Aj (W)
Selectad poliyy
Probability break (e)
Probubiliiy of break (5)
Figure 8: Fitting an I.V equilibrium curve on the empirical consumption rates of TPP (w; 7)- The plots are realization of the schematic diagram in Figure 2 using data from the MovieLens- IM experiment. (Leftt) Prediction with unbiased engagement predictions. (Right) User with extremely low engagement, for which the system tends to over-predict both cases, the LV policy optimiza- tion method selects the optimal policy p = O.
Results. Despite the difference between the true TPP and choice of model class, the LV policy optimization method successfully learned the optimal no- bwwbbbreakkwwwwee p for all users. Further detail provided by Figure 8, which illustrates the policy optimization steps under TPPCR (u: 2") for typical users with unbiased and biased engagement predictions. In both examples, the optimal points of both curves coincide, and the optimal policy is selected despite poor point-wise fit. Combined, these results show that our approach is "safe". in the sense that when breaking is sub-optimal, the learned breaking policy does not override the default policy.
in this case, is the policy
26


Reviews:
[{"review_id": "rRVMlc-2J9", "reviewer": "Reviewer_ggoW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper study an interesting problem that improves user engagement in sequential recommendations. Considering that current rules of taking a break are manually settled, this paper propose to learning a user dependent and stationary policy to decide when to take a break.\nLotka-Volterra dynamics is used to model two patterns of users, drive to consume and intrinsic interest. \n\nA learning algorithm based this dynamics is proposed and is validated on a semi-synthetic public data.", "summary_of_the_review": "Overall the paper is interesting and novel. Results are significant. However, some details need to clarified and the assumption needs to be justified.", "strength_and_weaknesses": "Strength:\n\n1.The problem of when to take a break is important and interesting for recommendation.\n\n2.The theoretical analysis of Lotka-Volterra dynamics is quite novel.\n\n3.The performance improvement as reported in Figure 3 is significant.\n\nWeakness:\n\n1.This paper aim to learn a policy that is independent of time step, which might limits the performance.\n\n2.This paper assumes that the behavior flows the Lotka-Volterra dynamics. Does the authors justify the assumption in real datasets?\n\n3.What is the detail of the evaluation metric? There is no engagement metric in MoiveLens.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8FCylVFci9K", "reviewer": "Reviewer_ZJ5Y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the authors study how recommenders could provide a personalized approach to recommending breaks to users that would still maximize long-term engagements.  The authors do this by assuming a particular dynamic model of user interest (Lotka-Volterra) and demonstrating how a policy could be learned for recommending breaks to users in conjunction with a recommender system.  They test this on simulated user dynamics on top of Movielens ratings data to demonstrate that their learned policy is more effective than alternative approaches.\n", "summary_of_the_review": "Overall, the paper is thorough in exploring a novel design space for recommender systems.  However, I believe significant additional research would be needed to demonstrate that their assumed dynamics model is reflective of real users and that their proposed approach does lead to higher long term user satisfaction.  As a result, I do not believe the paper is ready for publication at ICLR.\n", "strength_and_weaknesses": "S1. The question of how to better incorporate user well being into recommender design and how we can leverage long term models to do so is an interesting and important question to study.\nS2. Under the assumed model given, the paper is thorough in exploring the learning problem and demonstrating the efficacy of their learning approach.\n\nW1. The assumed model of user dynamic is highly novel - this is a strength in that it has not been used in the recommender literature before (as far as I know) but the paper does not actually explain or demonstrate why this particular model is a good one.  The experiments do show that it is flexible for learning even assuming a more complex user model, but alternative framings of the problem (e.g. from Kleinberg et al 2022) are not compared with and no empirical evidence is given that this fits or benefits real users at all.\n\nW2. The paper does not consider a lot of recent work and important aspects of recommender systems related to the actual items being recommended.  There is a large body of literature on diversity, dynamics of user interests within and across topics, additional signals of user feedback and satisfaction, leveraging surveys, etc.  In addition to not well contextualizing this work, this  both under-leverages state-of-the-art approaches for understanding user well being and is a limitation of the assumed user model, e.g. where user interest is unidimensinoal rather than evolving across many dimensions.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4bQL83J4jo", "reviewer": "Reviewer_VBoA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates how to promote digital well-being through individualized breaking policies. They propose a disciplined learning framework for responsible and sustainable optimization of long-term user engagement. For user behavior simulation, they utilize Lokta-Volterra (LV) dynamical systems to depict users as acting based on two balancing forces: drive to consume and intrinsic interest. The simulation is novel but lack of experimental verification. Finally, the paper proposes an efficient learning algorithm and evaluate it on semi-synthetic data.", "summary_of_the_review": "The paper offers a novel perspective on recommender systems, while the concrete design of user behavior model needs more experiments verification. ", "strength_and_weaknesses": "#Strength\n1. The paper studies recommendation systems as human-centric ecosystems, and take one step towards sustainable design. This provides a new perspective on viewing recommender systems.  \n2. The paper proposes to model user behavior as a dynamical system of Lotka-Volterra (LV) Predator-Prey equations to optimize engagement with breaks, where the effect of break policies is modeled by control variables. Such design is ingenious and innovative.\n\n#Weakness\n1. The paper needs to further explain the rationality of introducing LV dynamic systems. Under this model, the paper proposes two latent variables and analyzes their dynamical properties. There should be some online experimental results (e.g. A/B test) to verify the existence of such hidden variables and the dynamical properties of latent variables. Modeling the time-dependent relations as in Equation (2) seems not very convincing to me.\n2. The experiment description is not clear enough. \n3. The experimental results are not sufficient. It would be much better to conduct online evaluation or offline evaluation on more semi-synthetic datasets.\n4. The overall paper also needs an additional round of writing revision.\nGrammar Error:\n1)\tIn section 1 “To be preemtive” -> “To be preemptive”\n2)\tIn section 1  “with the persceince” -> ““with the prescience”\n3)\tIn section 1.1 “types of feedfback” ->  “types of feedback”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject