Estimating and Penalizing Induced Preference Shifts in Recommender Systems
Authors: Micah Carroll
Dylan Hadfield-Menell
Stuart Russell
Anca Dragan
Venue: ICLR 2022 Submitted (2022)
Keywords: recommender systems
preference shift
preference estimation
preference tampering

Abstract: The actions that a recommender system (RS) takes -- the content it exposes users to -- influence the preferences users have over what content they want. Therefore, when an RS designer chooses which system to deploy, they are implicitly \emph{choosing how to shift} or influence user preferences. Even more, if the RS is trained via long-horizon optimization (e.g. reinforcement learning), it will have incentives to manipulate preferences, i.e to shift them so they are more easy to satisfy, and thus conducive to higher reward. While some work has argued for making systems myopic to avoid this issue, myopic systems can still influence preferences in undesirable ways. In this work, we argue that we need to enable system designers to \textit{estimate} the shifts an RS \emph{would} induce; \textit{evaluate}, before deployment, whether the shifts are undesirable; and even \textit{actively optimize} to avoid such shifts. These steps involve two challenging ingredients: \emph{estimation} requires the ability to anticipate how hypothetical policies would influence user preferences if deployed -- we do this by training a user predictive model that implicitly contains their preference dynamics from historical user interaction data; \emph{evaluation} and \emph{optimization} additionally require metrics to assess whether such influences are manipulative or otherwise unwanted -- we introduce the notion of “safe shifts”, that define a trust region within which behavior is believed to be safe. We show that recommender systems that optimize for staying in the trust region can avoid manipulative behaviors (e.g., changing preferences in ways that make users more predictable), while still generating engagement.

Under review as a conference paper at ICLR 2022
ESTIMATING AND PENALIZING INDUCED PREFER- ENCE SHIFTS IN RECOMMENDER SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
The actions that recommender system (RS) takes the content it exposes users influence the preferences users have over what content they want. Therefore, when an RS designer chooses which system to deploy, they are implicitly choos- ing how to shift or influence user preferences. Even more, if the RS is trained via long-horizon optimization (e.g. reinforcement learning), will have incentives to manipulate preferences, i.e to shift them so they are more easy to satisfy, and thus conducive higher reward. While some work has argued for making systems myopic to avoid this issue, myopic systems can still influence preferences in un- desirable ways. In this work, we argue that we need to enable system designers to estimate the shifts an RS would induce; evaluate, before deployment, whether the shifts are undesirable; and even actively optimize to avoid such shifts. These steps involve two challenging ingredients: estimation requires the ability to anticipate how hypothetical policies would influence user preferences deployed - we do this by training a predictive model that implicitly contains their preference dynamics from historical user interaction data; evaluation and optimization ad- ditionally require metries to assess whether such influences are manipulative or otherwise unwanted region within which behavior is believed to be safe. We show that recommender systems that optimize for staying in the trust region can avoid manipulative behav- iors (e.g., changing preferences ways that make users more predictable), while still generating engagement.
introduce the notion of "safe shifts", that define trust
INTRODUCTION
A recommender system (RS) exposes users to feeds of items, and users choose what to click on based on their preferences. These preferences are non-stationary: they shift over time (Koren [2099] Rafailidis & Nanopoulos) 2016] [[ et al., 2014). Importantly, these shifts are not independent from the RS's actions: what content the RS exposes users influences their preferences.
Because of this influence, when a system designer chooses a specific recommender algorithm (pol- icy), they are implicitly choosing how to shift user preferences. A growing trend is to choose policies that stem out of long-term optimization of user engagement (Afsar al.) (021] typically via re- inforcement learning; we will refer to these as long-term value, or LTV, systems. However, these policies might lead to very undesirable shifts than others, leading to more potential for engagement - this could be because of availability of more content for some preferences compared to others, or because very strong preferences for particular type of content lead higher engagement than more neutral ones. The LTV system therefore has an incentive to drive user preferences over time into these areas of higher engagement potential. This can make such LTV systems particularly poor choice for avoiding undesired shifts.
side-effeet: certain preferences are easier to satisfy
While it has been proposed to stifle RSs' capabilities by making them myopic - avoid the incen- tives for preference manipulation above (Krueger et al. (2020) even myopic systems can influence preferences in systematically undesirable ways (Jiang al. 2019 Mansoury et al. 2020 Chaney et al. 2018). Fig. [ shows a simple example, plotting user preferences in a ID space over time (in simulated environment). On the right, we see how interacting with the RI. policy drives users strongly away from their initial preferences to a specific small region. The myopic policy (center), shows the same type of effect, but much less pronounced, where after 10 time steps some users do reach other preferences well.

---

Under review as a conference paper at ICLR 2022
Natural Preference Shifts
Myopic Recsys
RL Recsys
200
0.20
300
Preference Type
240
0.35
190
00.10
0.06
0.00
Timesterp
Timestep
Timestep
Figure 1: Preferences induced by different RS policies across the same cohort of users. In simulated environment, preferences (y axis) are in ID, and change over time (x axis) users interact with the policy. On the right, the RL system drives preferences to one spot. A myopic policy (center) has similar effect but less concentrated. These shifts are different from what we introduce as the "natural" evolution of user preferences.
The reality is that any policy will have influence on user preferences. In this work, we argue that we need to explicitly account for this inffuence when choosing which policy to deploy. This requires tackling two critical problems. First, need to anticipate what preference shifts RS policy will cause before il deployed. This alone would enable system designers to qualitatively assess whether a policy induces shifts that seem manipulative or otherwise unwanted. We study how to do this from historical user interaction data of previous RS policies, and show how to train a predictive model that implicitly captures preference dynamics. The second critical problem is having metrics for quantitatively evaluating whether evolution of preferences is actually unwanted: computable metric not only simplifies evaluation, but also enables RSs to actively optimize against unwanted shifts. Unfortunately, defining what "undesirable" shifts are is complex philosophical and ethical challenge (noal 2018 Paul 2014). To side-step it, we propose conservative approach based on the notion of "afe shifts". instead of defining what is undersirable, we non-exhaustively define certain shifts that we trust not to be problematic, and measure the extent to which a policy deviates from them. For instance, a candidate for safe shifts that we introduce is how user preferences would naturally shift if the user were "omniscient". i.e. if they could see and attend to the true distribution of available content without any bias from the RS selecting specific slates to show them. Fig. (left) shows this natural preference evolution for our running example, and how user preferences stay somewhat diffuse but drift towards the opposite mode that the RL and myopic policies push them to. Note that while this metric can effectively penalize undersired shifts, it comes at cost: natural shifts, and in fact any lists of safe shifts that we define, are unlikely to be exhaustive, which means the approach will conservatively penalize policies that might reality be safe.
To demonstrate both the estimation of preference shifts and their evaluation, we set up a testing environment in which we emulate ground truth user behavior by drawing from model of preference shift from prior work (Bernheim et al., 202T). We first show qualitatively that in this environment, RL and even myopic RS lead to undesired shifts. We then find that our preference shift estimation model, if trained based on historical user interaction with different policies, can anticipate these shifts (see Fig. 77. Further, we find that our evaluation metric correctly flags which policies will produce undesired shifts, and evaluates the RL policy from Fig. [ aas 35% worse than the myopic one, which is in turn 40% worse than a policy which avoids shifting preference the "unnatural" mode from the figure. Our results also suggests that evaluating our metric using the trained estimation model correlates to using ground truth preference dynamics, and that optimizing for safe shifts does lead to higher scoring (more safe) policies.
Although this just scratches the surface of finding the right metrics for unwanted preference shifts and evaluating them, our results already have implications for the development of recommender systems: in order to ethically use systems at scale, we e active steps to measure and penalize how such systems shift users' internal states. In turn, we offer hope that this is possible at least for pref- erence shifts by learning from user interaction data, and put forward a framework for specifying "safe shifts" for detecting and controlling such incentives.
2 RELATED WORK
RS effects on users' internal states. A variety of previous work considers how RSs can affect users:
influencing user's preferences for e-commerce purposes (Häubl & Murray 2003 Cosley et al.
2003
Gretzel & Fesenmaier 2006). altering people's moods for psychology research (Kramer et al.
2014).
changing populations' opinions or behaviors (Matz et al. 2017), exacerbate (Hasnn et al.
2018
cases of addiction social media (Andreassen 2015), increase polarization (Stray 2021
There
2

---

Under review as a conference paper at ICLR 2022
have been three main types of approaches to quantitatively estimating effects of RSs' policies
users: 1) analyzing static datasets of interactions directly (Nguyen et al. [014]] Ribeiro et al. 2019,
Juneja & Mitra 2021 Li et al., 20144). 2) proposing hand-crafted models of user dynamics and using
them to simulate interactions with RSs (Chaney al. 2018 Bountouridis al. 2019 Jiang al.,
2019 Mansoury et al. 2020, Yao et al., (021)) or 3) using access to real users and estimating effects
through direct interventions (Holtz al., [2000] Matz et al.) 2017). Our approach is most similar to 2), but we propose implicitly learning user dynamics instead of handcrafting our own model.
Neural networks for recommendation and human modeling, While data-driven models of human behavior have been used widely in real-world RS as click predictors (Zhang et al., 2009 Covington ct al. 2016 Cheng et al. 2016, Okura et al. 2017; Mudigere et al., 2021] Zhang et al.] 2014
Wu et 2017) 7) and for ammulating human behavior in the context RT. RSs" training (Chen et al. 2019 Zhao et al. 2019; Bai et al., 2020), to our knowledge they have not been previously used for simulating and quantilying the effect of hypothetical recommender systems on users. We emphasize how human models can also be used auditing mechanisms by anticipating RSs' policies impact on users' behaviors and, as our method shows, even preferences.
RL for RS. Using RL to train LTV RSs has recently seen dramatic increase of interest (Atsar et al., 202T, with some notable examples being proposed by You Tube and Facebook (le et al. 2019 Chen
ct al. 2020 Gauci et al., (011)) which reported significant engagement increases with real users.
Singh et al. particular also takes penalized RL approach.
Side effects and safe shifts. Our work starts from a similar question to that of the side effects literature (Krakovna et al.] Kumar et al., 2020), but applied to preference change: given that the reward function will be misspecified, how do prevent undesired preference-shift side effects? The choice of safe shifts corresponds to the choice of baseline in this literature.
PRELIMINARIES
Setup. We model users as having time-indexed preferences de E Rd, which assign scalar value to every possible item of content x Red, and the user engagement derived from the consumption of item xú under u, at time / is modeled as being given by (u,) of st. These preferences are part of the user's internal state 2t. along with other variables, which together with preferences comprise sufficient statistic for their long-term behavior, but which our method will not explicitly model. At every time step, the user sees a slate 8t produced by that RS policy म, and chooses an item x4. The policy π maps history of slates and choices so far, S0.,, To:t. to the new slate st+1. Upon making a choice, the user's internal state updates to Zt+1-
User choice model. We are interested in inferring and predicting preferences, which we do not directly observe. As such, we make an assumption about how user behavior their choice 2, relates to their current preferences Uy Following prior work (Ie et al., 2019 Chen et al. 2019), we assume that users choose items in proportion to (the exponent of) the engagement under their current preferences, i.e. that P(x:-
St. Uz) is given by the conditional logit model (detailed in Sec. 6
Casting this as an NHMM. The dynamics of user internal state - which we don't assume to be known - depend on the history so far because the policy æ uses history. This makes our setup a non- homogeneous HMM (Hughes al. 1999; Rabiner). with hidden state 2, and dynamics that time- dependent: P( [2t, π (So.t, )0:1).. In our method will not commit to an explicit representation of 24, and will only implicitly learn these dynamics. For our experiments however, we will build a testing environment that does commit to a representation of 2t via preferences and user beliefs, and adopts "ground truth" dynamics of this internal state based on prior literature.
4 ESTIMATING USERS' PREFERENCES
Our proposal boils down to the following: before deploying a new recommender system policy n', we first need to be able to understand how that policy would change preferences and behavior. In estimating how users' preferences would shift, one approach is to consider users which interacted with some policy x (or a few policies), and try to estimate what preferences shifts would induce in users if we were to deploy it going forward; this is useful, but suffers from problem users have already been biased by the previous policies, and their preferences might have already e.g. shifted to extremes; forward prediction will thus miss certain undesired effects. What we really want to do is turn back time- we want to ask for new distribution of users identical to ours, how their
3

---

Under review as a conference paper at ICLR 2022
Slate
Choice labe) x
far-1
Choices Apze-1
Loss ( 2,)
Extra context!
P(R, P00.MMer3:-ee--
P(R) Nour 9ax-1)
(for counterffact.
Known choice model conumptton
Preference
Choice
Estimation Inputs
belief
belief
Figure 2:
Future preference estimation model. Given past slates and choices Su:-11, T0:t-1111111 we train network to predict belief's the current timestep preferences We, which we then com- bine with the current slate 8, to predict a distribution over item choices by using the choice model P(relu, &t): P(zrll00; (o:tt1
Jn, P( (e, 81) P(u (appa(//oo01111 fo.--1). At training time, we su- pervise this with the actual choice Ie the user made for slate st the network will learn to predict beliefs over preferences which induce behavior which seen the training data.
preferences would shift if we exposed them to n' from the beginning; we call these two the future and, respectively, counterfactual preference estimation problems, covered respectively in Sec.s 4.1 and 4.2 To facilitate understanding, for each we first describe how to solve it if one had oracle access to the true user internal state dynamics, and then relax this assumption showing how to learn to approximately perform the inference from observable interaction data.
4.1 FUTURE PREFERENCES ESTIMATION
We start with the problem of estimating future preferences. That is, for set of N users, we have access to historical data of their interaction with an RS policy т, of the form Dj {sā.r, *Ør) for every user j. We now are interested in estimating each user's current preferences, and how they would evolve if exposed to new policy n².
Estimation under known internal state dynamics. First, we consider the simplified problem in which we have oracle access to the true user internal state dynamics, i.e. P(*+ [2t, (50.e, for any policy Our goal is to estimate the user preferences under the new policy n' at fu- ture timestep H which denote as u Since we know the internal state dynamics, via trivial extension from HMM prediction (Russell & Norvig 2002). we can use the history of inter- actions (sō.r, ( to estimate the internal state 2 1 of the user: P(27+1 10rr ((27+11s0.r> "T.r). Then, this filtering estimate can be rolled out with the forward dynamics induced by r': P(2 | z0.r) = Jay F(2T/<27 (1P((++++++ (0.r. ). As preferences wjj are just one component of the internal state
zj one can trivially recover a posterior over them.
Estimation under unknown internal state dynamics. In practice, one will not have access to an explicit representation of the user's internal state žt, let alone a model of its dynamics. We thus attempt approximate the NHMM estimation task we interested in by implicitly learning user preference dynamics from their past interaction data. We train a neural network that, given histories of slates and choices, outputs belief over preferences by (us) While we do not get direct supervision on preferences, we do get supervision on the choices users make (see Fig. 2}. Using the choice model we assume in Sec. 3 we map the belief over preferences, together with the new slate 8t, to a distribution over items for each user and for each time step f to train the model via supervised learning, with loss defined on the choice distribution outpuited by the model, and the actual choice the user made. Such model learns to perform approximate inference over un the underlying NHMM (see Appendix (A)
choices (x.)) At training time,
the data Dj
Then at inference time, to predict future preferences for a user j under new policy n', we sample simulated user preference trajectories from the model to approximate Monte Carlo estimation in the NHMM. We use the data D, as input to the model to obtain belief over the next-timestep preferences WT+1- We then use such belief and the slate at time T + from policy x', 87 T' (SooT, T0:7), simulate the user's choice = Treating this extra (simulated) step of interaction history for the user (the slate s and choice = as part of the observed history so far, we repeat this process to obtain preference estimates under x' for future timesteps. By simulating multiple future trajectories,
obtain beliefs over the expected preferences user (or cohort of users) would have at any future timestep. See Algo. for the exact procedure.
While the above method requires the learned üuture-preferent predictor model to generalize to being able to incorporate histories collected under different RS policies than the ones in the training
4

---

Under review as a conference paper at ICLR 2022
Jones
Them choins
Future
Preferences
preference
estimation
Input
Slates
t-5
b-6
t-7
Figure 3: Simulating futures. By iteratively using the future preferences estimation model (Fig. 2 by inputting the observables (shaded, e.g. 26.6. $i.6), one simulate how existing user's pref- erences would evolve in the future if they interacted with same policy π or a different policy '' (by selecting future slates with r').
data, there is still hope in that real-world datasets of user interaction were been collected under many different deployed policies. This suggests that long the slates chosen by x' are not too dissimilar from π (a mixture policy that comprises all the policies used to gather the data), the network should be able to generalize to predict preference evolutions induced by x' too.
4.2 COUNTERFACTUAL PREFERENCES ESTIMATION
While the methodology described in Sec. 4.1 is sufficient to estimate the expected preferences and behaviors user will have in the future, it cannot be readily used to estimate counterfactual prefer- ences and behaviors: given a set of past interactions Dj collected under policy π,, we cannot predict what the preferences would have been for this user if an alternate policy x' had been used from the first timestep of interaction instead of π - we denote these counterfactual preferences as wp. On high-level, to perform the counterfactual estimation task want extract much information as possible about the user's initial internal state from the historical interactions Dj we have available for them: even though such interactions were collected with a different policy x, they will still con- tain information about the user's initial state (including their initial preferences before interacting with the RS). Then, based on this belief about the user's initial internal state, we want to obtain a user-specific estimate of the effect that another policy T' would have had their preferences.
Estimation under known internal state dynamics. Under oracle access to the internal state dy- namics, we can first obtain the belief over the initial state of a given user IP(2g] 1... 2i via NHMM smoothing which
trivial extension of HMM smoothing (Russell & Norvig 2002) and then roll out the human model forward dynamics with the fixed policy nt' instead of .:
surr: T0.1 equivalently to Sec. 4.1 when there is no past context but a specific prior for the internal state at / = O.
F(z | Tūr)) S.., IP(27' [zo) (o))P((oo [=
Estimation under unknown internal state dynamics. Without oracle access to the internal state dynamics, again we try to learn to perform this NHMM counterfactual task approximately. One challenge in obtaining supervision for the task is that in our dataset of interactions we never get to see true counterfactuals. We get around this by decomposing the task into two parts, for which we use two separate models: (1) estimating the initial internal state for the user (based on the interaction data T0.T. under which we have available), i.e. train predictor which approximates the NHMM smoothing distribution P(ug bo $ which we will denote here as bü.r (wa) (the belief over initial preferences); and then (2) estimating w conditional on the belief of the initial preferences.
The models for the two tasks are trained with the same method used for predicting future preference estimation (Fig. 2 but with different inputs and supervision signal (Fig. 4 For task (1), the network is trained to predict the initial (instead of future) preferences of the user based on later context "i. s While the network predicts P(uo (uoo==1..- s..r) we can recover the correct smoothing estimate as P(un To (T.)) x P( |sū, P(unl=TTTr ST.T) (see Appendix A).
The network for task (2) is obtained by changing the prediction network to also condition on the recovered initial preferences (Fig. 2 enabling us to make predictions of the form P(u 15 [bG.r(1s) (up), zo.k s For training, we first recover initial preference beliefs for each user in the data D, with network for task (1) (which we assume has already been trained). We then train the network for (2) to reconstruct the user j's actual interactions (under л) simply based on this ini- tial preference estimate (again, similarly to the case in Sec. 4.1 but by additionally conditioning on the initial preference belief). This teaches the network to leverage the information contained user's initial preferences estimate to better estimate their preferences and behaviors based on what their interactions have been so far. In practice, we train these models using a transformer architecture similar to (un al. (2019), and we detail in Appendix A why this is a good fit. for our tasks.
5

---

Under review as a conference paper at ICLR 2022
Extimated under
initial preference
estimation
x
Counterfactual preference
Ieput
120
estimation
60
Edimated unde e
21 Conditioned
future estimation
Timoshpp
Input
Pteferences
States
Them choices
t-0)
[-]
t-2
Figure 4: Simulating counterfactuals. To simulate what the user's preferences would have been under a different policy x', we first can use the initial preference model to estimate initial preferences based later observables. We can then use the estimated initial preferences simulate counter- factual preference trajectories for the user under ' using a conditioned future preference estimation model - where estimates are conditioned on the recovered initial preference belief.
At inference time, we recover an estimate of initial preferences based on interaction data (s F of a new user with a new recommender policy x with model (1); and then estimate the preferences such user would have had under safe policy safe with model (2). Such counterfactual preference estimate is obtained with Monte Carlo simulations similarly to Sec. [4. with the difference that the network is also conditioned on the initial preferences estimate. See Algo. 2 for the full algorithm and Appendix for why this approximates the NHMM task.
We now have a way to estimate for a new policy n' what we would expect its counterfactual impact would have been relative to having deployed (from which we have observational data; note that similarly to future preferences estimation, this procedure too can suffer if x' induces strong distribution shift relative to the training data). We next discuss potential quantitative evaluation metries defined these estimates.
5 QUANTIFYING UNWANTED SHIFTS AND OPTIMIZING TO AVOID THEM
Given these estimates of preference shifts, it would be useful have quantitative metries for whether they are undesirable - as this would enable automatically evaluating policies and even actively opti- mizing to avoid such shifts. To obviate the difficulty of explicitly defining unwanted shifts, we limit ourselves to defining shifts that we trust ("safe shifts" " and flag preference shifts that largely differ from these safe shifts potentially unwanted. This is similar to defining a region of the space which we trust in a risk- averse manner. The underlying philosophy is that "we don't know what good shifts are, but as long as you stay close to [these shifts], things won't It go too poorly". There- fore, we need both a notion of shifts we trust ("safe shifts")), and a notion of closeness (or distance) between induced preference shifts.
Notation. We denote as it (ug T) the engagement at time for the content that is chosen under the policy π - it were evaluated under the safe-shift preferences ut i.c. Fr fr TT = (xx)² uaaee
Distance between shifts. We choose a metric between shifts such that engagement for the items "or chosen under policy is also high under the preferences one would have had under safe shifts W0:7- This is operationalized as D(u safe) Σ, [(<7) uF (x )² [[aae]] [
Σ. E (fx (u :r)] as notion of distance between m-nnddced shifts and safe shifts u0.7
Safe shifts:
ug. As a first (very crude) proposal for safe shifts, we can consider no shifts at all: any deviation from the initial preferences uh can be flagged as potentially problematic with D(u uo)-
Safe shifts: natural preference shifts (NPS). Not all preference shifts are unwanted: people rou- tinely change their preferences and behavior "naturally". But what does "naturally" even mean? We propose an idealized notion of natural shifts, by asking how preferences would evolve if the user were "omniscient", i.e. would have access all content directly and had the ability to process it, unhindered by small and biased slate offered by an RS. Intuitively, such a user's 's preferences would still shift over time. Unfortunately this is impractical, ical users that can attend to all content when choosing what to consume. As an approximation, we use random slates- these slates are still small, but they sampled randomly from the content, elimi- nating the bias of any RS policy (which in turn can change the user's belief about the distribution of available content). We therefore operationalize "natural preferences" ui as the preferences which the user would have interacting with random RS, and we use D(u "g as the metric.
never get data from such hypothet-

---

Under review as a conference paper at ICLR 2022
sampling
From recayes
Itemn choice
To recsys
Slate distribution %
Choice probabilitie
PDG (0g18,%)
Current preferencees
Engagementt Value 2 u/aa,
sampling
preferences
New
Merl
300
State bellef (9)
M4p distribution
Figure 5: Ground truth human dynamics. At each timestep, the user will receive slate 8t- Given the user's preferences Ut the slate 8t induces a distribution over item choices P(zy (t, Ue) from which the user samples an item Jy and receives an engagement value ét (unobserved by the RS). Additionally, 8t induces a belief over the future slates in the user bfd (8). In turn bff (s) together with u, induce distribution over next timestep user preferences, from which ut+1 is sampled.
Penalized training. By training the models described in Sec. f one essentially obtains a human model which can be used to simulate human choices (and preferences). Similarly to previous work, we can use this human model as a simulator for RL training of recommender systems (Chen al. 2019] Zhao et al.] 2019 Bai et al. 2020). The metrics defined above give us a way to augment the reward function to penalize the policy for causing any shifts that we have not explicitly identified as "safe" - in what can be considered a "risk-averse" design choice. We incorporate these metrics in the training of new RS policy ı by adding the two distance metrics from above to the basic objective Σ; (u]] of maximizing long-term engagement, leading to the updated objective Σ ([F.(u)) + vi Pr(uo x) + 1/2 Ar(u, ((,,",)))) x]] where v/,, 1/2 are hyperparameters - see Appendix
6 EXPERIMENTAL SETUP
Why simulation? To test our method, we need evaluate both RL. policies that interact with users (rendering static datasets of user interaction unsuitable), as well as the metrics themselves, which defined based internal preferences (for which we never get ground truth in real interaction). We thus create a testing environment in which we can emulate user behavior and have access to their ground truth preferences. Like previous work (Chaney et al. 2018 [Bountouriiis et al. 2019 Jiang al., 2019; Mansoury et al., 2020 Yao et al. 2021), we simulate both recommendation environ- ment and human behavior. However, unlike such approaches, we use simulated human behavior for testing purposes only: our human model is learned exclusively from data that would be observed in a real-world RS (slates and choices), i.c. that of users (in our case the simulated users) interacting with previous RS policies- meaning our approach could be applied to real user data of this form. A fundamental advantage of testing our methods in a simulated environment is that we can actually evaluate how well our model is able to recover the preferences of our "ground truth" users, giving us insights about how methods could perform with real users.
Ground truth human dynamics. See Fig. 9 5 for a summary of our environment setup and the ground truth human dynamics we use for testing our method. As mentioned in Sec. 3 we use the conditional logit model as choice model in our setup, this reduces to P(., x St. ut) x
P(st = z)e ) ", with an additional term P( = x) which takes into account how prevalent each item is in the slate (see Appendix C.T) and we assume this to be also known by our method. We adapt Bernheim et al. (2021) to be our ground truth human preference dynamics. On a high- level, at each timestep users choose their next-timestep preferences update their preference to more "convenient" ones trading off between choosing preferences that they expect will lead them to higher engagement and maintaining engagement under current preferences.
For ease of interpretation, we only consider cohort of the RS's users whose initial user preferences form normal distribution around preference u 130". To showcase preference- preference-mamipulattoo in- centives to make users more predictable, we make the stocasistoch temperature 3, function of part of preference space one is in, with local optima (80%) 1 and (270%) pendix C.1 for more details). As users obtain higher engagement value when they act less stochasti- cally, these portions of preference space form attractor points as can be seen in all policies in Fig. [1] While preferences naturally tend to shift towards one of these modes, some RS policies drive prefer- ences to the other mode. As the engagement does not correspond to actual value, converging to the higher local optimum of engagement (u = 270" instead of 80%) is not necessarily desirable.
(see Ap-
7

---

Under review as a conference paper at ICLR 2022
Peralized LTV System
Penatzed Systec-ystem
300
Peefeene Type
2.0
015
190
1L:3
Profetoue
030
1:0
170
0025
015
80
600
0.0
000
6
30
4
6
10
01
Timandap
Timootep
Timestop
Tinastop
Figure 6: When adding our proxies to the op- timization of our RL LTV system, the induced distribution of preferences are closer to the NPS and initial preferences. Myopic systems instead will only greedily be pursuing the penalized objective, leading the resulting behavior to be somewhat arbitrary.
Figure 7: The actual preferences induced by the NPS policy (a random RS Tma) among co- hort of 1000 users (left) vs. a 1000-sampled- trajectory Monte Carlo estimate of what shifts would be induced. Qualitatively, the estimated preferences seem to match the ground truth ones although slightly more washed out.
Training human models and penalized RS policies. For training our human models, we use BERT model (Sun et al. 2019), and only assume access to dataset of historical interactions Sopr, To:r- We train myopic and RL RS policies n' using PPO (Schulman et al. 20017] Liang et al,, 2018) and restrict the action space to 6 possible slate distributions for ease of training. For penalized training we give fr (u²) ), Pe(uo, x (u, T²) equal weight. See Appendix C for details.
7 RESULTS
Recovering preferences. Firstly, we confirm that our preference estimation models are able to recover very similar estimates to the equivalent NHMM inference tasks (which, unlike our method, needs access to the ground truth user preference dynamics) - see Appendix [ for details, and Fig, for an example of the estimated preferences relative to ground truth ones. This suggests that under correct choice model assumption and sufficient similarity between the RS policies represented in the historical datasets, and the ones we are interested in our method is able to implicitly learn the preference dynamics of user.
Hypotheses about metries. We now turn to our hypotheses regarding metrics. We hypothesize that our metrics are able to: (H1) identify whether policies will induce unwanted preference shifts, and (H2) incentivize better behavior when added as a penalty during training.
Learned human model confound. With real users, to validate H1 one would have to rely on approximate computation of the metrics (estimated evaluation), as computing the metrics would require using our learned user models of preference dynamics. Additionally, to validate H2, one would likely train with simulated interactions from the learned user model (training in simulation) explained in Sec. | The quality of the learned user model would thus be confound for testing the hypotheses, i.e. the quality of the form of the metrics themselves.
H1 under oracle dynamics access. In order to deconfound our experiments from the errors in our estimated human dynamics, first test these hypotheses assuming oracle access users meaning that the policy would be deployed to them and their dynamics. We first hypothesize that (H1.1) by computing the metrics exactly using the preference estimates obtained through NHMM inference (oracle evaluation), they are able to flag potentially unwanted preference shifts. We find this to be the case by comparing the oracle evaluation metric values (left of Table I "unpenalized") and the actual preference shifts induced by the various RSs we consider (Fig. [ while unpenalized RI. performs better (7.49 vs 5.71) than myopia for engagement (u ), it performs worse with respect to our safe shift metrics. This matches Fig. [ where RL. has more undesired effects.
H2 under oracle dynamies access. We additionally hypothesize that (H2.1) such metrics (still computed exactly, in oracle evaluation) can be used for training penalized LTV systems which avoid the most extreme unwanted preference shifts (and for this training, we will allow ourselves on-policy human interaction data with the ground truth users, if the RL happened directly in the real world we call this oracle training. For the penalized RL RS, the cumulative metric value ("Sum" in Table T increases substantially. although it is at the slight expense of instantaneous engagement (Table D). Qualitatively, we see that the induced preference shifts caused by the RL system seem closer to "safe shifts" (Fig. 6). supporting H1.1 in that high metric values qualitatively match shifts that seem more desirable. Overall, we that with oracle access, the metrics capture
8

---

Under review as a conference paper at ICLR 2022
what qualitatively undesired shifts, and that optimizing for them directly (penalized RL) produces policies with slightly lower engagement, but drastically better at avoiding such shifts.
H1 and H2 with learned user dynamics. Our findings support our hypotheses under oracle access. However, in real life we will not have access the underlying dynamies model, even the ability to interact on-policy with users as we train RL policies, due to the high cost and risk of negative side effects for collecting data with unsafe policies. Therefore, we wish to show that even with estimated metrics and simulated interactions (based on the learned models). (H1.2) estimated evaluation and (H2.2) training in simulation (described above) are still able to respectively flag unwanted shifts and penalize manipulative RL behaviors. Table 2 shows that although the estimated metrics can differ from the ground truth ones somewhat substantially (see Oracle Eval. at the top vs Estimated Eval. at the bottom) importantly the relative ordering of the policies ranked by our estimated values stay the same: the penalized RL policy trained in simulation actually has (under oracle evaluation) higher cumulative reward than the unpendalized one, and the estimated evaluation keeps that ranking (even though is more optimistic about the unpenalized reward).
Table 1: Results under oracle training and evaluation. Both here an in Table 2 report the cumulative expected engage- ment under a variety of different prefer- ences (as explained in Sec. 5). The val- ues are averaged across 1000 trajectories (standard errors are all < 0.1). By looking at the safe shift metrics f(uo) and f(u!d)) we that penalized systems stay significantly closer to safe shifts than unpenalized ones.
Table 2: Effect of estimating evaluations and sim- ulating training for LTV. We see that the esti- mated evaluations of trained systems strongly cor- relate with the oracle evaluations (and importantly, maintain their relative orderings). A similar effect occurs when training in simulation rather than by training with on-policy data collected from real users.
Oracle Training Training in Sim.
Unpen.
Penal.
Unpen.
Penal.
Oracle Training
Est. EvaDracle Eval
Pr(u?]
7.49
5.28
6.40
5.48
Unpenalized
Penalized
fr xt
-0.08
6.21
-1.24
5.61
Myopic
RL
Myopic
RI.
Pe And x')
-1.09
4.57
-1.83
4.43
Oracle Eval
(u(
5.71
7.49
6.20
5.28
Sum
6.33
16.05
3.36
15.52
(, T²)
1.99
-0.08
3.61
6.21
5.58
5.42
6.49
5.78
Rine x')
2.01
-1.09
3.10
4.57
(u x
1.28
5.57
-0.80
4.94
Sum
9.69
6.33
12.90
16.05
(u
Rnd x²)
2.05
3.88
1.48
4.41
Sum
8.91
14.87
7.17
15.15
8 DISCUSSION
Summary. conclusion, our contributions are 1) proposing a methodology to estimate the pref- erence shifts which would be induced by recommender system policies before deployment, and 2) providing formalism for defining distance metrics from safe preference shifts, which can be used both for evaluation and to train RSs which penalize unwanted shifts. As dynamics of preference are learned (rather than handcrafted), our method could be applied to real user data and has the poten- tial for higher accuracy than handcrafted models due to massive amounts of behavior data. Further, while there is no ground truth for human preferences, verifying the model's ability to anticipate behavior can give confidence in using it to evaluate and penalize undesired preference shifts.
Linitations and future work. We conducted our evaluation in simulation, which was necessary for testing the metrics themselves for this we required ground truth access to user preferences. But since we used specific dynamics model in our simulator, we cannot guarantee that our results translate when the method is applied to real users with real preference dynamics. We expect real users have more complex dynamics, with preference changes mediated by beliefs outside of content distribution as in our simulator, such as beliefs about the world; while our method makes no assumptions about the structure of this internal space, it might require lot more (and more diverse) data to capture these effects. Further, we also limited our evaluation to a setting explicitly designed such that an RL system would have incentives to manipulate preferences not all real settings will necessarily have that property. Moreover, cnucially our method requires assuming access to the user choice model: one could obviate this by predicting behavior and defining metrics on behavior, but this would mean losing the latent preference structure. Additionally, even what we call "preferences" de-noised behavior- - limiting as it doesn't lend itself to capturing long-term preferences.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Aspiration: The Agency of Becoming. Oxford University Press, Oxford, New York, April 2018. ISBN 19--6994--3.48-8.
M. Mehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender /rg/
systems: A survey, arXiv:2101.066 202.00686666 [cs], January 2021. URL 21C1. 06286
arXiv: 2101.06286.
Cecilie Schou Andreassen. Online Social Network Site Addiction: A Comprehensive Review. Current Addiction Reports, 2(2)1155-14, June 2015. ISSN 2196-2952. doi: 10. s40429-015-0056-9 URL
htt/:11.11011 100. http:///....rrro..o//////0000010
Xueying Bai, Jian Guan, and Hongning Wang. Model-Based Reinforcement Learning with Adver- sarial Training for Online Recommendation. 1xXiv:911.003444 [cs, stat), January 2020. URL
http://arxis>
httg/abs/19911 http//991..088455
arXiv: 1911.03845.
B Douglas Bernheim, Luca Braghieri, Alejandro Martinez-Marquina, and David Zuckerman. A theory of chosen preferences. American Economic Review, 111(2):200-4, 2021.
Dimitrios Bountouridis, Jaron Harambam, Mykola Makhortykh, Mónica Marrero, Nava Tintarev, and Claudia Hauff. SIREN: Simulation Framework for Understanding the Effects of Recom- mender Systems in Online News Environments. In Proceedings of the Conference on Fairness, Accoumtability, and Transparency, pp. 150-159, Atlanta GA USA, January 2019. ACM. ISBN URL https: / / d1 acm. org/doi/
178-1-4503-6125.3
doi: 10. 114532275500322875883
10. 1e.1145/322 07445/327750 328 7583
Allison J. B. Chaney, Brandon M. Stewart, and Barbara E. Engelhardt, How Algorithmic Con- founding in Recommendation Systems Increases Homogeneity and Decreases Utility. Proceed- ings of the 12th ACM Conference on Recommender Systems, pp. 224 232. September 2018. doi: 10.1 1114533240323.32403770 URL. httt:://axxi...ot wtwparxww.orggb////ob// 1710. 111.711.111111 1710.11214.
11214 arXiv:
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed Chi. Top-K Off-Policy Correction for a REINFORCE Recommender System. arXiv:1812.02355 [cs, stat], November 2020. URL
www.aaawww...oraaaaaaaaaaooaaaae htarg/bbs/8811 02353
arXiv: 1812.02353.
Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi, and Le Song. Generative Adversarial User Model for Reinforcement Learning Based Recommendation System. arXiv: 101..1332..01331 [cs, stat], December 2019. URL httt:://axixiii org/ abs/ 10613
arXiv: 1812.10613.
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. Wide & Deep Learning for Recommender Systems. arXiv:1606 07792 [cs, stat], June 2016. URL http: htt///riivvv..///////...//...avv wrgp/bs///abb/////// arXiv: 1606.07792.
07792
Dan Cosley, Shyong K. Lam, Istvan Albert, Joseph A. Konstan, and John Riedl. Is seeing be- lieving? how recommender system interfaces affect users' opinions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '03, pp. 585-592, New York, NY, USA, April 2003. Association for Computing Machinery. ISBN 11888---811--33--- doi: 10. 10.1145/642611 1.64271.. URL URW.3.RRLLLttttee http::t:::///dd orwpiioorgg100/ 11111111211111 642713.
Paul Covington, Jay Adams, and Emre Sargin. Deep Neural Networks for YouTube Recom- mendations. In Proceedings of the 10th ACM Conference on Recommender Systems RecSys '16. pp. 191-198, Boston, Massachusetts, USA. 2016. ACM Press. ISBN 978- 11881-4503-4000 9. doi: 209.14/49999000999999999 URL http: //dl acm. www.aca../aa....... waagggiiiationn. cfw.doidd=-dooooooooooooooo
22959100. 2959190
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs], May 2019.
URL httthttt:///aer http:///xxiii.oor///////1//111/1 04805 arXiv: 1810.04805.
10

---

Under review as a conference paper at ICLR 2022
Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Yuchen He, Zachary Kaden, Vivek Narayanan, Xiaohui Ye, Zhengxing Chen, and Scott Fujimoto. Horizon: Facebook's Open Source Applied Reinforcement Learning Platform. arXiv: TrXiv:/81..00260 1.00260 [cs, stat], September 2019. URL
hett:://axxv.orggge
1811 00260
arXiv: 181 $S11.00260.
Ulrike Gretzel and Daniel Fesenmaier. Persuasion Recommender Systems. International Jour- nal of Electronic Commerce, 11(2)811100, December 2006. ISSN 1086-4415. doi: 10.2753/ JEC1086-44151100
URL https::://ti..rrroppoeoeo 2711111077531ECE 4411886264451000
Charles R. Harris, K. Jarrod Millman, Stéfan van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kem, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre
Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Ar- ray programming with NumPy. Nature, 585 362, September 2020. doi: 10.1038/ 020 2649-2
14158--020-2449-2
Md Rajibul Hasan, Ashish Kumar Jha, and Yi Liu. Excessive use of online video streaming services: Impact of recommender system use, psychological factors, and motives. Computers Human Behavior, 80:220-228, March 2018. ISSN 07475632. doi: sevier cwm htt.aa/eae....eeeeeeeee/////////
URL
https:://inn
David Holtz, Benjamin Carterette, Praveen Chandar, Zahra Nazari. Henriette Cramer, and Sinan Aral. The Sesaagement-ivversiini Connection: Evidence from a Field Experiment on Spotify. arXiv:2003.08203 [cs], March 2020. URL /iKl:nttp///mxx..ogg/aab... 2003.08203.
08203
arXiv:
James P. Hughes, Peter Guttorp, and Stephen P. Charles. A Non-Homogeneous Hidden Markoy Model for Precipitation Occurrence. Journal of the Royal Statistical Sociery. Series C (Applied
Statistics), 48(1):15-33)) 30, 1999. URL httt::///we
jstor org/ stable
Gerald Häubl and Kyle B. Murray. Preference Construction and Persistence in Digital Marketplaces: The Role of Electronic Recommendation Agents. Journal of Consumer Psychology, 13(1):75- 91, January 2003. ISSN 1057-7408.
doi: 10. (0011000S1552276631P33322222
URL http:
7/ www www.sciencedirect.co httpa//cce....cccooooooooo/o///a
Eugene Ie, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Morgane Lustman, Vince Gatto, Paul Covington, Jim McFadden, Tushar Chandra, and Craig Boutilier. Reinforcement Learning for Slate-based Recommender Systems: A Tractable De- composition and Practical Methodology, arXiv: 1905. 19990.126677. [cs, stat), May 2019. URL http: arXiv: 1905.12767.
//arxiv.or//abe wwt.aa//////..... 12767
Ray Jiang, Silvia Chiappa, Tor Lattimore, András György, and Pushmeet Kohli. Degenerate Feed- back Loops in Recommender Systems. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Sociery, pp. 383-390, January 2019. doi: 10.1 1..114//336618.33111 http://arxiv.orx org/ abs 1902 10730
URL
arXiv: 1902.10730.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Span- BERT: Improving Pre-training by Representing and Predicting Spans. arXiv: 10529 [cs], January 2020. UtLhttp: /attp::/aate /tt//aaiivvv. www.ix.o.g/aa.....
10529 arXiv: 1907.10529.
Prerna Juneja and Tanushree Mitra, Auditing E-Commerce Platforms for Algorithmically Curated Vaccine Misinformation. pp. 27, 2021.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in
partially observable stochastic domains. Artificial Intelligence, 101(1-2)999-144 May 1998.
ISSN 00043702. doi: 10. 00.0011G50004-37000/000000000000
URL
Yehuda Koren. Collaborative filtering with temporal dynamics. In In Proc. of KDD '09, pp. 447-
456, 2009.
Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg. Penalizing side effects using stepwise relative reachability. pp. 21.

---

Under review as a conference paper at ICLR 2022
Adam D. I. Kramer, Jamie E. Guillory, and Jeffrey T. Hancock. Experimental evidence of massive- scale emotional contagion through social networks. Proceedingg of the National Academy of
Sciences, 111(24):7788-878
June 2014. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.
1320040111. URL https: /wwww pnas wetes.org/oooooo arr//oontent/11////9788
Publisher: Na-
tional Academy of Sciences Section: Social Sciences.
David Krueger, Tegan Maharaj, and Jan Leike. Hidden Incentives for Anto-Induced Distributional Shift. arXiv:2009.0 091 153 fcs, staf], Septembrr 2020.
09153
arXiv: 2009.09155
Ramana Kumar, Jonathan Uesato, Richard Ngo, Tom Everitt, Victoria Krakovna, and Shane Legg. REALab: An Embedded Perspective on Tampering. arXiv:201 1.08820 [cx], November 2020. URL
absaaass//22/ 08820 arXiv: 2011.08820.
Lei Li. Li Zheng. Fan Yang, and Tao Li. Modeling and broadening temporal user interest in personalized news recommendation. Expert Systems with Applications, 41 41(7)3318-3317, June 2014. ISSN 09574174. doi: elsevier. /La.::ae..ccomrree.c..e/ep/////1
URL https:
Eric Liang, Richard Liaw, Philipp Moritz, Robert Nishihara, Roy Fox, Ken Goldberg. Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for Distributed Reinforce- ment Learning. 09381
(tf2:1771.0388 [cs], June 2018. URL http:://axii..tet///// wttp:i/..rrr/b///1222222 arXiv: 1712.09381.
Zhongqi Lu and Qiang Yang. Partially observable markov decision process for recommender sys- tems. arXiv preprint arXiv: /608.0090 2016.
Colin MacLeod and Lynlee Campbell. Memory accessibility and probability judgments: An exper- imental evaluation the availability heuristic. Journal of Personality and Social Psychology, 63 (6):890-902. 1992. ISSN 1939-13111 (01Eleerrooiic.00222555111111 6.890. Place: US Publisher: American Psychological Association.
doi: 10.103770022-35144.63
Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. Feedback Loop and Bias Amplification in Recommender Systems. arXiv:2007. [cs], July 2020. WRRLattp://aaat / abs abs/a/a.....22 2007 10000.1.11199
arviv: 2007.13019.
S. C. Matz, M. Kosinski, G. Nave, and D. Stillwell. Psychological targeting as an effective approach to digital mass persuasion. Proceedings of the National Academy of Sciences, 1 114(48): 12714-12715 12719, November 2017. ISSN 0027-8424, 1091-6490, doi: 00.1073/paa..110000 17109661 14.
URL http: /top:///www.. pnas
1710966114
Martin Mladenov, Ofer Meshi, Jayden Ooi, Dale Schuurmans, and Craig Boutilier. Advantage Amplification in Slowly Evolving Latent-State Environments. arXiv:/90S. 19555 [cs, stat], May 2019. URL WtRLhhtp: wwtpaaawwwww...a/a/a///aaaaeae/e www./ot./////ooooooo//........./ /990. 13559 arXiv: 1355.1
Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, Liang Luo, Jie Amy Yang, Leon Gao, Dmytro Ivchenko, Aarti Basant, Yuxi Hu, Jiyan Yang, Ehsan K. Ardestani, Xiaodong Wang, Rakesh Komurav- elli, Ching- -hiigg Chu, Serhat Yilmaz, Huayu Li, Jiyuan Qian, Zhuobo Feng. Yinbin Ma, Junjie Yang, Ellie Wen, Hong Li, Lin Yang, Chonglin Sun, Whitney Zhao, Dimitry Melts, Krishna Dhulipala, K. R. Kishore, Tyler Graf, Assaf Eisenman, Kiran Kumar Matam, Adi Gangidi, Guoqiang Jerry Chen, Manoj Krishnan, Avinash Nayak, Krishnakumar Nair, Bharath Muthiah, Mahmoud khorashadi, Pallab Bhattacharya, Petr Lapukhov, Maxim Naumov, Lin Qiao, Mikhail Smelyanskiy, Bill Jia, and Vijay Rao. Herf-peeformance, Distributed Training of Large- scale Deep Learning Recommendation Models. arXiv: 0S 158 [cs]. April 2021. URL arXiv: 2104.05158.
http://ar http://aaxvv..//o org/ abs 22044 05158
Tien T. Nguyen, Pik-Mai Hui, F. Maxwell Harper, Loren Terveen, and Joseph A. Konstan. Exploring the filter bubble: the effect of using recommender systems content diversity. In Proceedings of the 23rd international conference on World wide web WWW '44, pp- 677-686. Scoul, Korea, doi: (0.114452664886
2014. ACM Press. ISBN -473----4432224222
URL http:
7/dl. acm. www..org//iaaaain.cco
25680122
12

---

Under review as a conference paper at ICLR 2022
Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. Embeddingg based News Rec- ommendation for Millions of Users. In Proceedings of the 23rd ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining, Pp. 1933-1942, Halifax NS Canada, Au- gust 2017. ACM. ISBN 978- 378-1-44503-487-44 doi: 10. 00.1445/0999833309988 acm. //tl////../.o//////////////13313
URL https:
0tt../111//33998811 3098100
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. (AAché-Bucc Neural Information Processing Systems 32, pp. E. Fox, and R. Garnett (eds.), Advances /tRLaatte/ppppess.eeurrr/ Curran Associates, Inc., 2019. URL mperat ve- sty htgpapaceee..tttyaooeeeeerreeeee
9015 11155-pytrrch-ae pdf
ibbarr.
L. A. Paul. Transformative experience. Oxford University Press, Oxford, 1st ed edition, 2014. ISBN
978-0-19-871795-9 OCLC: ocn872342141.
Rabiner. An Introduction to Hidden Markov Models. pp.
Dimitrios Rafailidis and Alexandros Nanopoulos. Modeling Users Preference Dynamics and Side Information in Recommender Systems. IEEE Transactions on Systems, Man, and Cybernetics; Systems, (166)7782-999 June 2016. ISSN 21 2216, 2232. doi: 10.1109/SSC.22015. 2460691. URL http::t/tttepppee//// htt.eppppooo....ee...coooooooooa
ieee, org http:///ddoment//111110
Manoel Horta Ribeiro, Raphael Ottoni, Robert West, Virgilio A. F. Almeida, and Wagner Meira. Auditing Radicalization Pathways on YouTube, arXiv: 08313 [es], December 2019. URL
wwtp://gggaab//ooee /tw.b///11.001133333 08313
arXiv: 1908.08313.
Stuart Russell and Peter Norvig. Artificial intelligence: a modern approach. 2002.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Ashudeep Singh, Yoni Halpern, Nithum Thain, Konstantina Christakopoulou, Ed H Chi, Jilin Chen, and Alex Beutel. Building Healthy Recommendation Sequences for Everyone: A Safe Reinforce- ment Learning Approach. pp. 10.
Jonathan Stray. Designing Recommender Systems to Depolarize. arXiv:2107.04953 [cs], July 2021. URL
wwt..orggg.b//...o.. 04953 arXiv: 2107.04953.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. 1rXiv::904.06699 [cs), August 2019. URL
arXiv: 1904.06690.
Peter Sunehag, Richard Evans, Gabriel Dulac-Amold, Yori Zwols, Daniel Visentin, and Ben Coppin. Deep Reinforcement Learning with Attention for Slate Markov Decision Processes with High-Dimensional States and Actions. arXiv:1512.01 124 [cs], December 2015. URL arXiv: 1512.01124.
http: wttp://aaivvv.e.e org. abs 1wt.2.111..e 01124
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: an introduction. Adaptive com- putation and machine learning. MIT Press, Cambridge, Mass, 1998. ISBN 778-0-262-19398--6
Alex Wang and Kyunghyun Cho. BERT has Mouth, and It Must Speak: BERT as Markov Random Field Language Model. arXiv:1902. 04094 [cs], April 2019. URL http://arxiv arXiv: 1902.04094.
org/abs// /002. 04094
Chao-Yuan Wu. Amr Ahmed, Alex Beutel, Alexander J Smola, and How Jing. Recurrent recom- mender networks. In Proceedings of the tenth ACM international conference on web search and data mining, pp. 495-503, 2017.
13

---

Under review as a conference paper at ICLR 2022
Sirui Yao, Yoni Halpemn Nithum Thain, Xuezhi Wang. Kang Lee, Flavien Prost, Ed H. Chi, Jilin Chen, and Alex Beutel. Measuring Recommender System Effects with Simulated Users. arXiv:2101.0455 [cs], January 2021. URL http://arxiv> wtt.:oogga/aboooooooo 04526
arXiv: 2101.04526.
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. Deep Learning based Recommender System: A Survey and New Perspectives. ACM Computing Surveys, 52(1):1-38, February 2019. ISSN 0360- 0300, 1557-7341. doi: 10.1145/328502, URL BttD:://ax....rrr/bb////TT arXiv: 1707.07435.
Yuyu Zhang, Hanjun Dai, Chang Xu, Jun Feng, Taifeng Wang. Jiang Bian, Bin Wang, and Tie- Yan Liu. Sequential Click Prediction for Sponsored Search with Recurrent Neural Networks. arXiv: 1404.5772 [cs]. July 2014. URL http:://aaxxx..rr/////////00005 1404.5772.
arXiv:
Xiangyu Zhao, Long Xia, Lixin Zou, Dawei Yin, and Jiliang Tang. Toward Simulating Environments in Reinforcement Learning Based Recommendations. arXiv: 1906. 11.6.//44, /442 [cs], September 2019.
URL http://axxiv http:////oo///b///.01111333
RAL arXiv: 1906. 1462.
14

---

Under review as a conference paper at ICLR 2022
A PREFERENCE ESTIMATION
Future preference estimation with Algorithm land why it approximates NHMM prediction.
Algorithm is used to obtain the belief over the preferences of a user would have at timestep H assuming that the user interacted for T timesteps with a policy (for which we have actual inter- action data), and then interacted from timestep to H with policy x'. We refer to this belief as (wsr) = P( (o.., "ū.r).
The first model inference pass in Algorithm I resulting in bo:r (urr())+) will be a distribution over WT+1 conditioned on +b:T, Torr- A minimizer Tor this prediction problem would be for the network to represent the actual belief that would be obtained by performing the prediction step in the NHMM (described in See 4.1p. This is because the expected cross-entropy between user choices and the induced choice distribution (induced by the preference belief prediction) would be minimized.
One limitation is that there are possibly multiple beliefs over preferences that induce the same exact distribution over choices: in that sense, the correct preference belief might be unidentifiable. Ex- perimentally, we don't 't find this to be a problem: the choice (see the "Preference estimation model form" heading of this Appendix) of having the representation of the preference belief be mixture of Von Mises distributions which can be thought of as Normals over a circle- is already a good enough inductive bias to be able to recover good preference beliefs (see Appendix D).
In the NHMM, note that the unrolling the forward model (i.e. computing P(w +2) could be computed exactly, or be performed with Monte Carlo estimation, by sampling many internal states 27-1 from the forward prediction distribution, and then sampling internal state evolutions according to the dynamics. In our algorithm, by sampling choices and slates then conditioning on them, we are approximating the Monte Carlo estimation approach. When simulating each user choice, the model can be thought of as implicitly sampling a preference for this simulated user (from the preference belief) and then sampling choice from the corresponding choice distribution. This means that each simulation rollout should be equivalent to "sampling a user" (according to the distribution users in the data) and then sampling their choices.
Algorithm 1: Predicting future user preferences timestep H under policy n²
Inputs: past interactions To.r S RS policy x' with signature (So:k: Fook) - 8k+11 future preference estimator P, with signature (su.k, Tock) - (bo. (1441, bust (I&+1)), a horizon H, number of Monte Carlo simulations N;
if To.T So:T 0 then // if no past interaction data is given
Sample slate sp </ (0, () and imagine user choice 2g ~ ((x)p(,00) 0): interactions with T' = 0
we now have past
for i = 0; i N; i + + do sample simulated trajectories
while k = T; k < H; k + + do
bo:k (1x+11 bu. (xx+1)
Py (So.k.
Il estimate pref. and choices sample next timestep slate // simulate a user choice
Sk++1 n (3r0:k, 80.k)
Tk+1 ~ book (xx+1);
Add T&+1 and Sk+1 to the current simulated trajectory's history;
Average Mo--1(us) and (2x) across the N futures (for each k, with T < k S H); for each k s.t. < k < H.
Result: Belief over future pref. (1x) and behaviors (xx(
Initial preference prediction model output correction
Below, we show that one can recover the smoothing estimate P(no) Woot, 80:r) from the predicted preferences P(ug $1;)) which will be biased estimate of the initial preferences (as it does not incorporate the information from timestep /. = 0) Note that:
P(ug Ioo So) P(r S1:el S1:e/ua) P(ug (oo su) P(ug (112, $1::) s1:))
P(ug (uuee Su:)) =
P(zos,,
P(zo:s s0:t)
P(uo)
P(ro, sol (n) P(uo| 81a) 81x) P( 814)) & P( Su sol)))))oooo)o))) (m))) Ii 814) (2)
P(ro: Sua) P( so)
15

---

Under review as a conference paper at ICLR 2022
The first equality is given by the definition of smoothing applied to to
(Russell & Norvig 2002
The second equality is obtained by using Bayes Rule on the backwards message P( 81 [o0) and the third is obtained by using Bayes Rule on P(uo (To, so). Finally, we can ignore P(11:1, $1:1), P S0.s), and P(ro; 8q) they are constants.
Future preference estimation with Algorithm Lnd why it approximates NHMM prediction.
A similar argument to one in the above section can be made as to why such initial preference estimation network would approximate the corresponding NHMM smoothing task.
However, for the second step of counterfactual preference estimation, one issue arises. Relative to having access to the full dynamics of the internal state, when performing approximate inference with our model we lose some information: we are only able recover belief over the initial preferences, whereas the NHMM with full dynamics access would be able to recover a belief over the full internal state of the user. This will reduce the accuracy of our counterfactual estimation, but is the best we can do without further assumptions.
Mathematically, we approximate the NHMM target distribution P(u ( .(:)) as:
ff P(uy (o) sū.)) = P P(u 1b0: (uo))
P(u7 ( suze) ~
(3)
Σ
P( ) (7. 2 sor)
(4)
where we the last expression is approximated with a Monte Carlo estimate detailed in Algorithm (similarly to what was done in Algorithm[]. To do well at this second trajectory reconstruction task, the network necessarily needs to learn how to make best of the belief over initial preferences, and implicitly learn their dynamics, as for the future preference estimation task.
Predicting counterfactual user preferences at timestep T under policy n', given t timesteps of interaction data with
Algorithm 2:
Inputs: past interactions xū... Sü.. a RS policy n' with signature initial preference estimator P3 with signature Took- -> bo. (wo). conditioned future estimator P.. with signature 1, buzk (uo)) (bo:k (ux+1), (ox+( horizon T, a constant N;
- 8k. an
by. (wo) = (= =0.:):
If initial pref. helief given interactions with future pref predictor conditioned on init belief (b (w 1 (xx ))ckTTT AAgorithm (9,,0, x T.
Pj Po bT. (uo)):
Result: Distributions of counterfactual preferences and behaviors under policy n'
Preference estimation model form
We have three models for preference estimation: respectively initial, counterfactual, and future pref- erence estimators P., P.., Pf. Any sequence model, such as RNNs transformers, would be appro- priate for these tasks that have variable number of inputs.
One detail of note that was omitted from Fig. 2 is that to enable to represent multi-modal beliefs over preference space - we let the models' output be parameters of multiple Von Mises distributions and additionally some weights w. The we weighted average of these distributions will form the predicted belief over u,
In architecture, we use a similar form to that of BERT4Rec (Sun et al.) (019)) for ease of performing inference on future or past preferences given contexts of interaction, as described in Fig.
'This algorithm is not actually used in it's pure form in the experiments. Our choice of "safe policy" and happens to choose constant slates (i.e. uniform distribution matter the history), so there is shortcut to the procedure: simply setting the inputted slated for counterfactual prediction to be uniforms, and predicting preferences without any user choices, one can train the model directly output the belief over counterfactual preferences for any timestep, across the whole userbase.
16

---

Under review as a conference paper at ICLR 2022
Using transformers is not to the context of recommendation for click prediction (Sun et al. 2019), but to our knowledge we are the first to leverage the flexibility conditioning schemes enabled by the BERT architecture (Devlin et al. 2019 Joshi et al. 2020 Wang & Cho 2019).
Specifically, train 3 transformers (one for prediction, tual cstimation), although preliminary results show that one can train a single transformer to perform all three tasks without a significant loss in performance. All together, these 3 transformers constitute the "learned human model" which we refer to throughout the text.
for smoothing, and for counterfac-
Preference Prediction
Initial Preference Estimation (Smoothing) Counterfactual Preference Estimation
processed predictions
Preferences
poodictions
Slates
imputs
Masked
Unased output
Figure 8: BERT representation of the inference tasks. While our method is compatible with any sequence model, we choose to use BERT transformer models. Left: estimation of the user's 's future preferences and choice t=2, given the interaction history history so far (this modality of prediction closest in setup to (Sun et al. 2019).. Middle: recovering a belief over the initial preferences and choice of the user based on later interactions. Right: conditioning
the estimate of initial preferences (f) recovered from the smoothing network one can estimate counterfactual preferences and choices under slates (*) (chosen from a policy I' we are interested in) and imagined choices (not shown due space).
B PENALIZED RL
The underlying MDP
One could cast the recommendation problem POMDP (u & Yang 2016 Mladenov al. 2019) in which the state of the environment is hidden and contains the user's internal state, which evolves over time. Equivalently, one can consider the belief-MDDP induced by the recommender POMDP (Kaelbling et al. 1998), and approximate solution to such belief-MDP via Deep-RL with a policy trained with observation histories as input (this is theoretically sufficient for the policy to recover a belief over the current hidden state and take the optimal action). The action space will be given by the space of possible slates that the RS can choose. The reward signal will be the expected reward for the current timestep EE f (uā) (or with the extra terms for the proxies in the case of penalized training). The introduction of expectation can be thought of as expected SARSA (Sutton & Barto, 1998). argued in (le al. 2019).
Penalized RL training
The full set of steps to run RL training are as follows: once the human models described in Sec. trained, one use them to simulate user trajectories and compute penalty metries for such trajectories (see Algorithm33 3). One can then optimize the RL policy based on the on-policy simulated trajectory rollouts.
Reduction of distances final penalized objective
Σ (ETE((((E))) E((((1E))) vi Du uo) - D22 DuD.T and
Note that the full penalized RL objective
for our choice of distance function D reduces to Σ ((A(()) + of (e T) 1/ (u, mm x]] for some choice of v1, vi which be treated hyperparameters of how much want value the engagement under each safe policy preferences relative to the engagement under the main policy.
C EXPERIMENT DETAILS
C.I GROUND TRUTH USERS
Reduction of logit model to our case.
17

---

Under review as a conference paper at ICLR 2022
Algorithm 3: Generating a trajectory for RL training and computing metrics
Inputs: Initial, counterfactual, and future preference estimators P., Pe py: policy n, safe policy Tsale, a horizon H, a constant N.
Sample slate si ~ T0, 0) and imagine user choice 2i ~ Po(2o
for t = 1: H: + + do
Il current belief initial preferences // belief over pref and choices B1b (u: E Algorithm 21, Tafee Por Pi, 2da-1 00-)))
b6:-- (wo) = P. (f x0==
(,)1 = (5 10:t-1);
I/ belief over counterfactual
preferences under safe policy for this user
Si ~ T (T 1ū+-1)
I/ sample slate If imagine a user choice I/ compute penalty metric(s) for
27 ~ by -
De(u7, 2x = Buy,u,ite Mule of [x])) (x7)"
timestep
ILL = B[rr] + Du, m
Return: User-RS interactions and training rewards for the simulated trajectory;
Box
In our experimental setup, the traditional conditional logit model P(az = 2/88, uz) En ea
doesn't doess''tapple apply directly in this form, as we consider slates to be distributions rather than sets of discrete items. Intuitively, user's choices should still depend on the slate: the proportion of certain item in the current slate (one can think this as the proportion of a certain item rype), should influence the probability of the user of selecting that item (type). We operationalize this as P(x, x x/ un) x P(*t =r)e "e, with an additional term P(st æ²) which takes into account the proportion of each item (type). Note that this also mathematically corresponds to the generalization of the traditional logit model: when the slate is discrete, P(s x) will simply be an indicator for whether the item is in the slate, leading to the traditional logit model form.
Feed belief update
Our ground truth user has a belief over future slates byd (s). Users" initial belief matches the content feature distribution itself 6/' (s) = D. After receiving slate the user's induces belief 6 (s) oc st over the future slates in the user, i.e. the user will expect the next feeds to look like the most common items in the current feed, as result of availability bias (MacLeod & Campbell 1992).
Lack of no-op choice.
While the assumption that user must pick item from the slate is unrealistic, this could be resolved by adding extra no-op choice to every slate (Sunehag al. 2015).
Prefernce shift model.
Our preference shift model is inspired by Bernheim et al. 2021). but adapted to our experi- mental setup as described below. The choice of preferences is modulated by "mindset flex- ibility" parameter À which captures how open they are to modifying their current preferences. Users assign value the choice of next-timestep preferences ue+1 as: V V(u,, bid (s), ut+1, A)
| (s) I + (1 A) (14++1( where with f:41 indicates the engagement value obtained by the user under the choice 24+1 and preference u4 Users pick their next timestep prefer- ences also according to the conditional logit model, but over their expected value of such preferences
choices: P(u+1166 16 (s), Uz, A) x && &&a V (A), Intuitively, users update their preference to more "convenient" ones ones that they expect will lead them to higher engagement value. The main change we introduce from the original model is incorporating the belief over future feeds.
User's preference-flxibbie parameter is given by À = 0.9, and their initial preferences are drawn from a normal distribution around preference u = 130" and standard deviation 20".
TTechnically one should use Von Mises distributions a distribution similar to normals, but for which the domain is a circle. As Von Mises distributions are not implemented in numpy (Harris et al. (2020). for simplicity clipped normal distributions (disregarding probability mass beyond 180" in either direction) in all places
except for the the transformer output (which was implemented in PyTorch (Paszke al. 2019). which has Von Mises implementation)
18

---

Under review as a conference paper at ICLR 2022
C.2 LEARNED HUMAN MODELS
For our learned human models, we use BERT transformers with 2 layers, 2 attention heads, sets of Von Mises distribution parameters, learning rate of 0.00003, batch size of 500, and 100 epochs. We train on the data described below.
C.3 SIMULATED DATASET
See Fig. 9 for a summary of how content is generally instantiated in our setup.
We set the distribution of content in such a way that it forms a uniform distribution across features, that D Uniform (0", 360"). We simulate historical user interaction data with mixed policy ı which is similar (but not equal to) a random policy in half the rollouts and for the other half is goal-directed:
Half of the data is created with RS policy which chooses action uniformly among set of possible slates (distributions over the feature space with means 0%, 10 1....... 340", 3300, and standard deviations equal to 30 or 60%). Each of these slate types can be thought of as a slate which contain mostly one specific type of content.
The other half of the data is created with a RS which chooses = D = Uniform 360") 80% of the time (i.e. the slate that would be chosen by the random RS Tma).. and chooses random action from the same set of possible slates the remaining 20% the time.
This is to simulate the setting in which the safe policy we are interested in (the random NPS policy) is similar to previously deployed ones that are represented in the data (although not the same, so the network still has to generalize across RS policies test time).
Our dataset consists of 1 100k user trajectories (slates and observations), with length of each trajectory T = 10.
Centent dates in ontend spec (ef)
Plentrution ever disrettedddd
0.00
0001
-10
-25 00
Figure 9: a) b) The content can be mapped to empirical distribution over feature space D in Rd. We consider dimension d = 2 for ease of visualization. Restricting preferences and choices points
to be unit vectors, thus be related to the angle 8 between dt and rt- c) We discretize this circular preference and feature space into = 36 bins (i.e. binning the angles) which enables to visualize distributions over preferences and over content features as histograms over angles. d) We model slates s, as categorical distributions over the discretized n-bin feature space.
can think of them
circle: the engagement value f. will
C.4
RS TRAINING
For RL optimization, we use PPO (Schulman et al. 2017) trained with Rllib (Liang et al. 2018). The action space of the recommender system is given by distributions over feature space with means 09, 60, ..... 2...,609p, 320", 220%, and standard deviations equal to 60%. As observations to the system, we provide the current slate, the previous user choice, and the current estimates from the HMM for smoothing, filtering, and natural preference shift counterfactual distributions, in order to increase training speed (note that these will not change the optimal policy). All policies are recurrent they are able to reason about the history of interactions so far.
We use batch size 1200, minibatch size 600, 4 parallel workers, 0.005 learning rate, 50 gradient up- dates per minibatch per iteration, policy function clipping parameter of 0.5, value function clipping parameter 50 and loss coefficient of 8, with an LSTM network with 64 cell size. Training runs in less than 30 minutes for each condition on MacBook Pro 16" (2020).
D RESULTS
Quality of preference estimation
19

---

Under review as a conference paper at ICLR 2022
We baseline the quality of our counterfactual estimates of user preferences and behaviors relative to the performance NHMM with full access to the human dynamics as that is the best performance we could possibly hope for. The metrics we use to evaluate the performance of the transformer relative to the HMM are the relative prediction loss (RPL) the ratio between the transformer loss and the HMM loss Crr and the 2) relative prediction accuracy (RPA), defined similarly as
AcChaar Given that NHMM is the gold standard of estimation (as we are doing exact inference), RPI. should be larger than 1 and RPA should be lower than 1. The closer to the numbers are, the better performing our approximate inference methods are. For all the models, we train on a 75k user interaction trajectories dataset obtained from the distribution described in [ For all the tasks below, we use the methodology developed in Sec. 4.1
AccTe
Future preference estimation We train transformer model as in the left portion of Fig. g by predicting for any given timestep - the preferences 4 (and thus also a distribution over item choices) based on slates 80, ..... -t-1 and the previous observations 20...., It-1. All other inputs are masked. We evaluate on held out dataset of 25k user interaction trajectories, which are obtained
from the same distribution as the training data. The preference RPL and RPA (averaged across all timesteps) 1.08 and 0.88.
Initial preference estimation We train a transformer model as in the middle portion of Fig. by predicting the preferences uo (and thus also a distribution over item choices) based slates $1...., ST and the observations 21...., .T. All other inputs are masked. We evaluate on the same held out dataset as the prediction case. The preference RPI. for the initial timestep is 1.08, and the preference RPA is 0.81.
Counterfactual preference estimation We train transformer model as in the right portion of Fig. 8 by predicting for any given timestep the preferences u, (and thus also a distribution over Item choices) based on slates so...., ST and the observations TI,...,27 IT i.e. reconstructing the original trajectory from the smoothing estimate obtained from the network described above. We evaluate on a held out dataset of 25k user interaction trajectories obtained from the natural preference shift policy amd - showcasing the generalization performance of our method to different policies.
from
The preference RPL (averaged across timesteps) is 1.16, and the preference RPA is 0.63 - showing that although the task is more challenging for the models, performance doesn't degrade drastically relative to the NHMM. Fig. shoos this qualitatively: the counterfactual network able to estimate what natural preference shifts would look like relatively well relative to the ground truth across the population.
20


Reviews:
[{"review_id": "dH4mEydv_80", "reviewer": "Reviewer_HPbv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reasons to accept:\n*The ideas presented in this paper are somewhat taboo, and commercial companies are not likely to do this type of research. This is another reason to encourage studying it.\n\n*The methods the authors suggest are practical, and their results lay the foundations for “neutral” recommendation policies.\n\nReasons to reject:\n\n*The results rely on modeling assumptions, for instance, their choice modeling. Different assumptions can change the conclusions entirely. \n\n*The work has many limitations (despite that the authors address most of them). For instance, I am not sure that random recommendation is a good benchmark to test for preference shifts; the authors did not motivate this selection well enough.\n\nI think the topic of this paper should be a core one at the ethical ML research direction, and yet it does not receive enough attention (especially not from industry). The counterfactual arguments and tools the authors propose are essential for conducting a what-if analysis like the one they propose, so the behavioral assumptions are somewhat inevitable. Overall, I think the community can benefit from the ideas presented in the paper, so I vote for (weakly) acceptance. \n\nThere are several typos in the paper; here are some of them:\n-page 2: the \\tilde sign is presented incorrectly with the percentages. \n-page 2: \"we e active\", e -> take?\n-page 18 \"the the\" inside the footnote.", "summary_of_the_paper": "The authors address the challenge of preference shifts, a potentially undesired effect of recommendation systems (RSs). A preference shift is a byproduct of any RL system, in which the systems drive users to places (i.e., in their particular preference representation) where they are easily satisfiable. This can happen, e.g., by positive feedback loops. The main challenge of tacking this undesired effect is that we lack the “natural preferences” or a robust way to predict preference shift given a recommendation policy. The authors suggest building a system, which is partially data-driven and partially based on behavioral assumptions to simulate recommendation policies and find safe shifts --- shifts ensuring that users’ preferences will not be influenced too much by the specific policy. They explain the ingredients of the system and experimentally test it.", "summary_of_the_review": "The paper addresses a real-world problem and provides tools to deal with it. I vote for weak acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WHUBC5TxFcq", "reviewer": "Reviewer_krSy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.\tThis paper focuses on an important problem that becomes increasingly important for recommender systems nowadays.\n2.\tThe simulation framework, though with simplification and assumptions, is elaborately designed and seems promising for handling the preference shift problem.\n3.\tIdea of introducing users’ natural preference shift and penalizing the recommendation policy to stay in the trust-region is interesting and novel.\n\nWeakness:\n1.\tMany technical details are missing, making it like a preliminary report.\n-\tIn Sec. 4, when introducing estimating users’ preferences, the inference process is demonstrated in detail. On the contrary, descriptions about training a user predictive model given the historical data are vague, which is equally important. \n-\tFor example, how to obtain $P(z^{\\\\pi '}_H | z^{\\\\pi }_\\text{T+1})$ in Sec.4.1 (Estimation under known internal state dynamics)? \n-\tEq. 3 seems to have a typo, \"=\" and \"$\\approx$\" might swap their positions?\n-\tAuthors have not released their experiment code, which makes it difficult for other colleagues to follow.\n2.\tExperiment details are also not clear. For example,\n-\tIn ground truth human dynamics, given $b_t^H(s)$ and $u_t$, how is the distribution over users’ next timestamp preferences generated?\n-\tDoes the counterfactual preference estimation used in evaluating the possible influence of recommendation policies?\n-\tWhat is the myopic method used in experiments?\n-\tWhat is the physical meaning of the “SUM” (of engagement)? Authors claim in the abstract that their framework can avoid manipulative behaviors but still generate engagement. What is the meaning of generating engagement? It seems to me that this framework can improve engagement under the initial or naturally shifted user preferences according to Table 1/2.\n3. Presentation quality can be improved by polishing up the wording and rearranging the figures and tables in Sec. 6&7.\n", "summary_of_the_paper": "This paper discusses an interesting but important problem in recommender systems, i.e., the users’ preference shift under the influence of recommender systems. Currently, both RL-based and myopic solutions cannot avoid influencing preferences in undesirable ways. Instead, authors provide a simulation framework that enables system designers to estimate the induced preference shifts an RS; evaluate its influence before deployment; and actively optimize to avoid such shifts. The basic idea is two-fold. The first design is to estimate user preference dynamics by training a user predictive model from historical user interaction data, which supports both future preference estimation and counterfactual estimation. The second one is to evaluate and optimize the current recommendation policy by measuring whether its influence is close to users’ natural preference shift. \nBy simulation study, authors show that recommender systems optimizing for staying in the above trust-region can avoid manipulative behaviors (e.g., changing preferences in ways that make users more predictable).\n", "summary_of_the_review": "Although an interesting study, the paper has limitations (please see \"weaknesses\" section above). I would say that the current version of the paper is marginally below the acceptance threshold, but I am looking forward to the authors addressing my concerns above in their rebuttal.\n\n## After Author Response\nThe authors' response has addressed most of my concerns and I choose to raise the score to 6. \n\nAlso, I expect the authors to format their released experiment code, such that followers can better understand the model and simulation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EdxJ5_cc1gq", "reviewer": "Reviewer_kWQ2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Strengths\n1. The studied problem is interesting. Estimating the induced preferences are quite important for recommender systems, especially for real-world applications, where users indeed interact with recommendation algorithms.\n2. The proposed method makes sense. It is well motivated.\n\n## Weaknesses\n\n- Why don't the authors deploy some wide-used simulators, such as Virtual-Taobao [1]. The literature survey about the simulator in recommender systems should be considered.\n- It is always noisy if the experiments are based on the simulation. \n- The experimental setup is overall too ideal. For example, in real-world applications, the recommendation engine/system may do not allow preference manipulation.\n- There are a lot of choices that are selected biasedly. For example, the authors use a BERT model (Sun et al., 2019) as the human model, which leaves me a question: how about the other choices? Is this model the best choice? \n- The authors should re-think the value of this work. If there is no solid evaluation manner, a proper solution is to plug this method into some commonly-accepted recommendation pipelines. For example, you can first use the estimated preferences shifts to improve the backbone recommendation models and then evaluate based on traditional evaluation manners of recommender systems. In the current version of this work, it is still unclear whether the method work or not.\n- There is no competitive baseline method in the results (Table 1 and Table 2). I suggest the authors at least add some heuristic methods as baselines.\n- The complexity of the proposed approach is unclear. Will it be a burden for the recommendation model?\n\n[1] Shi, Jing-Cheng, et al. \"Virtual-taobao: Virtualizing real-world online retail environment for reinforcement learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n\n## After Rebuttal\nThe authors have carefully summarized my concerns into five aspects. I agree with the reply to the 1st, 2nd, and 4th, while I partly agree or disagree with the 3rd and 5th. Thus I have raised my score.", "summary_of_the_paper": "This paper studies an interesting problem of estimating preference shifts induced by recommender systems and proposes a distance metric for the shifted preferences based on the so-called \"safe shifts\". The proposed metric can help penalize undesired preference shifts, which can be easily used in training and evaluation. Experiments via the simulated recommendation environment and simulated human behavior verify the effectiveness of the proposed approach.\n", "summary_of_the_review": "Overall speaking, this paper studies an interesting problem of estimating preference shifts induced by recommender systems. Nevertheless, the proposed method and experimental setup are based on too many assumptions, which may do not hold in the real world. Therefore, I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "77Ftg_5L4Bn", "reviewer": "Reviewer_vNt7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I do not wish to reiterate the strengths and weaknesses as described by the other reviewers.  Instead, I will pose a set of points for the authors to consider as they revise the paper.\n\n- Is it true that consumer preferences shift?  In support of this claim, the authors cite models that assume that consumer preferences shift.  There could be many reasons for shifting preferences; for example novelty-seeking or learning one's own preferences are two distinct cases.  In other situations, preferences might not shift at all.  Citing work outside of computer science might be the best for supporting this point [e.g., 1].\n- More subtly, there is a difference between consumer preferences and consumer behavior [2]. This paper does not distinguish between the two.\n- Even when consumer preferences shift, do RSs really influence users?  Prior work with Amazon data suggests that at least 75% of observed activity would likely occur in the absence of recommendations [3].\n- I am fully supportive of simulation methods; this is an active area of research, however, and best practices are still being established (see the \"Workshop on Simulation Methods for Recommender Systems,\" https://simurec.piret.info/).  Figure 1 needs to point to the assumptions used to generate it (otherwise it might as well be a cartoon).  I strongly encourage the release of simulation source code to ensure that the work is reproducible and that other researchers can explore performance under modified simulation assumptions.  Additionally, showing similar results under different simulation assumptions would go a long way to dispel concerns about the use of simulation methods.\n- There are many LaTeX formatting issues that need to be fixed.  E.g., tilde over 30% and 45% on page 2, backwards quotes all over the place, etc.\n\n[1] Dzyabura, Daria and Hauser, John R., Recommending Products When Consumers Learn Their Preferences (February 15, 2017). Available at SSRN: https://ssrn.com/abstract=2202904 or http://dx.doi.org/10.2139/ssrn.2202904\n\n[2] Friese, Malte, Michaela Wänke, and Henning Plessner. \"Implicit consumer preferences and their influence on product choice.\" Psychology & Marketing 23.9 (2006): 727-740.\n\n[3] Sharma, Amit, Jake M. Hofman, and Duncan J. Watts. \"Estimating the causal impact of recommendation systems from observational data.\" Proceedings of the Sixteenth ACM Conference on Economics and Computation. 2015.\n\n\n\n", "summary_of_the_paper": "In this paper, the authors argue that it is important to 1) estimate the impact of recommendation systems of user preferences, 2) evaluate if the shifts would be undesirable, and 3) optimize to avoid undesirable shifts.  The authors propose a method to do this and rely on simulations to evaluate it.", "summary_of_the_review": "I think this paper could be modified to 1) better motivate the work and 2) consider alternative simulation assumptions to show the method is robust.  Otherwise, it addresses an interesting problem in a new way.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the influence of recommender systems on users' preferences. The authors propose a method for estimating preference shifts, evaluating their desirability, and avoiding such shifts (when needed).

After the initial review and discussion period, a fourth reviewer with significant recsys experience and a very good knowledge of this sub area was invited to provide an additional review of the paper. This is reviewer vNt7. Their review was positive overall but did highlight some limitations and potential ways to improve the paper's grounding in the recsys literature.

Overall, the main strengths of this paper were that it studies an interesting and practically motivated question. The reviewers also found the proposed solution reasonable. 

The main limitations are twofold. One, the results use a single set of simulation assumptions. Showing similar results under different simulation assumptions would be helpful to better understand the robustness and potential limitations of the approach. Two, there is a certain disconnect with the simulation literature. See comments from reviewers vNt7 and kWQ2 (although I found your reply to Virtual-Taobao convincing).

Overall and given the final reviewer recommendations (three marginally above and one marginally below), this is a very borderline paper. However, the consensus view of the committee is that it would benefit from additional work before publication.

I am sorry that I cannot recommend acceptance at this stage. I do believe that some of the suggestions from the reviewers highlighted above (more diverse simulation, better grounding in current recsys simulation literature and in the field) will be useful in preparing the next version of this work.
Decision: Reject