Year: 2021

Abstract:
The actions that a recommender system (RS) takes – the content it exposes users to – influence the preferences users have over what content they want. Therefore, when an RS designer chooses which system to deploy, they are implicitly choosing how to shift or influence user preferences. Even more, if the RS is trained via long-horizon optimization (e.g. reinforcement learning), it will have incentives to manipulate preferences, i.e to shift them so they are more easy to satisfy, and thus conducive to higher reward. While some work has argued for making systems myopic to avoid this issue, myopic systems can still influence preferences in undesirable ways. In this work, we argue that we need to enable system designers to estimate the shifts an RS would induce; evaluate, before deployment, whether the shifts are undesirable; and even actively optimize to avoid such shifts. These steps involve two challenging ingredients: estimation requires the ability to anticipate how hypothetical policies would influence user preferences if deployed – we do this by training a user predictive model that implicitly contains their preference dynamics from historical user interaction data; evaluation and optimization additionally require metrics to assess whether such influences are manipulative or otherwise unwanted – we introduce the notion of “safe shifts”, that define a trust region within which behavior is believed to be safe. We show that recommender systems that optimize for staying in the trust region can avoid manipulative behaviors (e.g., changing preferences in ways that make users more predictable), while still generating engagement.

text: The actions that a recommender system (RS) takes – the content it exposes users to – influence the preferences users have over what content they want. Therefore, when an RS designer chooses which system to deploy, they are implicitly choosing how to shift or influence user preferences. Even more, if the RS is trained via long-horizon optimization (e.g. reinforcement learning), it will have incentives to manipulate preferences, i.e to shift them so they are more easy to satisfy, and thus conducive to higher reward. While some work has argued for making systems myopic to avoid this issue, myopic systems can still influence preferences in undesirable ways. In this work, we argue that we need to enable system designers to estimate the shifts an RS would induce; evaluate, before deployment, whether the shifts are undesirable; and even actively optimize to avoid such shifts. These steps involve two challenging ingredients: estimation requires the ability to anticipate how hypothetical policies would influence user preferences if deployed – we do this by training a user predictive model that implicitly contains their preference dynamics from historical user interaction data; evaluation and optimization additionally require metrics to assess whether such influences are manipulative or otherwise unwanted – we introduce the notion of “safe shifts”, that define a trust region within which behavior is believed to be safe. We show that recommender systems that optimize for staying in the trust region can avoid manipulative behaviors (e.g., changing preferences in ways that make users more predictable), while still generating engagement.

heading: 1 INTRODUCTION
text: A recommender system (RS) exposes users to feeds of items, and users choose what to click on based on their preferences. These preferences are non-stationary: they shift over time (Koren, 2009; Rafailidis & Nanopoulos, 2016; Li et al., 2014) . Importantly, these shifts are not independent from the RS’s actions: what content the RS exposes users to influences their preferences.
Because of this influence, when a system designer chooses a specific recommender algorithm (policy), they are implicitly choosing how to shift user preferences. A growing trend is to choose policies that stem out of long-term optimization of user engagement (Afsar et al., 2021) – typically via reinforcement learning; we will refer to these as long-term value, or LTV, systems. However, these policies might lead to very undesirable shifts as a side-effect: certain preferences are easier to satisfy than others, leading to more potential for engagement – this could be because of availability of more content for some preferences compared to others, or because very strong preferences for a particular type of content lead to higher engagement than more neutral ones. The LTV system therefore has an incentive to drive user preferences over time into these areas of higher engagement potential. This can make such LTV systems a particularly poor choice for avoiding undesired shifts.
While it has been proposed to stifle RSs’ capabilities by making them myopic – to avoid the incentives for preference manipulation above (Krueger et al., 2020) – even myopic systems can influence preferences in systematically undesirable ways (Jiang et al., 2019; Mansoury et al., 2020; Chaney et al., 2018). Fig. 1 shows a simple example, plotting user preferences in a 1D space over time (in a simulated environment). On the right, we see how interacting with the RL policy drives users strongly away from their initial preferences to a specific small region. The myopic policy (center), shows the same type of effect, but much less pronounced, where after 10 time steps some users do reach other preferences as well.
The reality is that any policy will have influence on user preferences. In this work, we argue that we need to explicitly account for this influence when choosing which policy to deploy. This requires tackling two critical problems. First, we need to anticipate what preference shifts an RS policy will cause before it is deployed. This alone would enable system designers to qualitatively assess whether a policy induces shifts that seem manipulative or otherwise unwanted. We study how to do this from historical user interaction data of previous RS policies, and show how to train a predictive model that implicitly captures preference dynamics. The second critical problem is having metrics for quantitatively evaluating whether an evolution of preferences is actually unwanted: a computable metric not only simplifies evaluation, but also enables RSs to actively optimize against unwanted shifts. Unfortunately, defining what “undesirable“ shifts are is a complex philosophical and ethical challenge (noa, 2018; Paul, 2014). To side-step it, we propose a conservative approach based on the notion of ”safe shifts”: instead of defining what is undersirable, we non-exhaustively define certain shifts that we trust not to be problematic, and measure the extent to which a policy deviates from them. For instance, a candidate for safe shifts that we introduce is how user preferences would naturally shift if the user were ”omniscient”, i.e. if they could see and attend to the true distribution of available content without any bias from the RS selecting specific slates to show them. Fig. 1 (left) shows this natural preference evolution for our running example, and how user preferences stay somewhat diffuse but drift towards the opposite mode that the RL and myopic policies push them to. Note that while this metric can effectively penalize undersired shifts, it comes at a cost: natural shifts, and in fact any lists of safe shifts that we define, are unlikely to be exhaustive, which means the approach will conservatively penalize policies that might in reality be safe.
To demonstrate both the estimation of preference shifts and their evaluation, we set up a testing environment in which we emulate ground truth user behavior by drawing from a model of preference shift from prior work (Bernheim et al., 2021). We first show qualitatively that in this environment, RL and even myopic RS lead to undesired shifts. We then find that our preference shift estimation model, if trained based on historical user interaction with different policies, can anticipate these shifts (see Fig. 7). Further, we find that our evaluation metric correctly flags which policies will produce undesired shifts, and evaluates the RL policy from Fig. 1 as 3̃5% worse than the myopic one, which is in turn 4̃0% worse than a policy which avoids shifting preference to the ”unnatural” mode from the figure. Our results also suggests that evaluating our metric using the trained estimation model correlates to using ground truth preference dynamics, and that optimizing for safe shifts does lead to higher scoring (more safe) policies.
Although this just scratches the surface of finding the right metrics for unwanted preference shifts and evaluating them, our results already have implications for the development of recommender systems: in order to ethically use systems at scale, we e active steps to measure and penalize how such systems shift users’ internal states. In turn, we offer hope that this is possible – at least for preference shifts – by learning from user interaction data, and put forward a framework for specifying “safe shifts“ for detecting and controlling such incentives.

heading: 2 RELATED WORK
text: RS effects on users’ internal states. A variety of previous work considers how RSs can affect users: influencing user’s preferences for e-commerce purposes (Häubl & Murray, 2003; Cosley et al., 2003; Gretzel & Fesenmaier, 2006), altering people’s moods for psychology research (Kramer et al., 2014), changing populations’ opinions or behaviors (Matz et al., 2017), exacerbate (Hasan et al., 2018) cases of addiction to social media (Andreassen, 2015), or increase polarization (Stray, 2021). There
have been three main types of approaches to quantitatively estimating effects of RSs’ policies on users: 1) analyzing static datasets of interactions directly (Nguyen et al., 2014; Ribeiro et al., 2019; Juneja & Mitra, 2021; Li et al., 2014), 2) proposing hand-crafted models of user dynamics and using them to simulate interactions with RSs (Chaney et al., 2018; Bountouridis et al., 2019; Jiang et al., 2019; Mansoury et al., 2020; Yao et al., 2021), or 3) using access to real users and estimating effects through direct interventions (Holtz et al., 2020; Matz et al., 2017). Our approach is most similar to 2), but we propose implicitly learning user dynamics instead of handcrafting our own model.
Neural networks for recommendation and human modeling. While data-driven models of human behavior have been used widely in real-world RS as click predictors (Zhang et al., 2019; Covington et al., 2016; Cheng et al., 2016; Okura et al., 2017; Mudigere et al., 2021; Zhang et al., 2014; Wu et al., 2017) and for simulating human behavior in the context RL RSs’ training (Chen et al., 2019; Zhao et al., 2019; Bai et al., 2020), to our knowledge they have not been previously used for simulating and quantifying the effect of hypothetical recommender systems on users. We emphasize how human models can also be used as auditing mechanisms by anticipating RSs’ policies impact on users’ behaviors and, as our method shows, even preferences.
RL for RS. Using RL to train LTV RSs has recently seen a dramatic increase of interest (Afsar et al., 2021), with some notable examples being proposed by YouTube and Facebook (Ie et al., 2019; Chen et al., 2020; Gauci et al., 2019) – which reported significant engagement increases with real users. Singh et al. in particular also takes a penalized RL approach.
Side effects and safe shifts. Our work starts from a similar question to that of the side effects literature (Krakovna et al.; Kumar et al., 2020), but applied to preference change: given that the reward function will be misspecified, how do we prevent undesired preference-shift side effects? The choice of safe shifts corresponds to the choice of baseline in this literature.

heading: 3 PRELIMINARIES
text: Setup. We model users as having time-indexed preferences ut ∈ Rd, which assign a scalar value to every possible item of content x ∈ Rd. and the user engagement derived from the consumption of item xt under ut at time t is modeled as being given by r̂t(ut) = uTt xt. These preferences are part of the user’s internal state zt, along with other variables, which together with preferences comprise a sufficient statistic for their long-term behavior, but which our method will not explicitly model. At every time step, the user sees a slate st produced by that RS policy π, and chooses an item xt. The policy π maps history of slates and choices so far, s0:t, x0:t, to the new slate st+1. Upon making a choice, the user’s internal state updates to zt+1.
User choice model. We are interested in inferring and predicting preferences, which we do not directly observe. As such, we make an assumption about how user behavior – their choice xt – relates to their current preferences ut. Following prior work (Ie et al., 2019; Chen et al., 2019), we assume that users choose items in proportion to (the exponent of) the engagement under their current preferences, i.e. that P (xt = x|st, ut) is given by the conditional logit model (detailed in Sec. 6). Casting this as an NHMM. The dynamics of user internal state – which we don’t assume to be known – depend on the history so far because the policy π uses history. This makes our setup a nonhomogeneous HMM (Hughes et al., 1999; Rabiner), with hidden state zt and dynamics that timedependent: P ( zt+1|zt, π(s0:t, x0:t) ) . In our method will not commit to an explicit representation of zt, and will only implicitly learn these dynamics. For our experiments however, we will build a testing environment that does commit to a representation of zt via preferences and user beliefs, and adopts a ”ground truth” dynamics of this internal state based on prior literature.

heading: 4 ESTIMATING USERS’ PREFERENCES
text: Our proposal boils down to the following: before deploying a new recommender system policy π′, we first need to be able to understand how that policy would change preferences and behavior. In estimating how users’ preferences would shift, one approach is to consider users which interacted with some policy π (or a few policies), and try to estimate what preferences shifts π′ would induce in users if we were to deploy it going forward; this is useful, but suffers from a problem – users have already been biased by the previous policies, and their preferences might have already e.g. shifted to extremes; forward prediction will thus miss certain undesired effects. What we really want to do is turn back time – we want to ask for a new distribution of users identical to ours, how their
preferences would shift if we exposed them to π′ from the beginning; we call these two the future and, respectively, counterfactual preference estimation problems, covered respectively in Sec.s 4.1 and 4.2. To facilitate understanding, for each we first describe how to solve it if one had oracle access to the true user internal state dynamics, and then relax this assumption showing how to learn to approximately perform the inference from observable interaction data.

heading: 4.1 FUTURE PREFERENCES ESTIMATION
text: We start with the problem of estimating future preferences. That is, for a set of N users, we have access to historical data of their interaction with an RS policy π, of the form Dj = {sπ0:T , xπ0:T } for every user j. We now are interested in estimating each user’s current preferences, and how they would evolve if exposed to a new policy π′.
Estimation under known internal state dynamics. First, we consider the simplified problem in which we have oracle access to the true user internal state dynamics, i.e. P ( zt+1|zt, π(s0:t, x0:t) ) for any policy π. Our goal is to estimate the user preferences under the new policy π′ at a future timestep H – which we denote as uπ ′
H . Since we know the user internal state dynamics, via a trivial extension from HMM prediction (Russell & Norvig, 2002), we can use the history of interactions (sπ0:T , x π 0:T ) to estimate the internal state zT+1 of the user: P (z π T+1|sπ0:T , xπ0:T ). Then, this filtering estimate can be rolled out with the forward dynamics induced by π′: P(zπ′H |sπ0:T , xπ0:T ) =∫ zπT P(zπ′H |zπT+1)P(zπT+1|sπ0:T , xπ0:T ). As preferences uπ ′ H are just one component of the internal state zπ ′
H , one can trivially recover a posterior over them.
Estimation under unknown internal state dynamics. In practice, one will not have access to an explicit representation of the user’s internal state zt, let alone a model of its dynamics. We thus attempt to approximate the NHMM estimation task we are interested in by implicitly learning user preference dynamics from their past interaction data. We train a neural network that, given histories of slates and choices, outputs a belief over preferences bπ0:T (ut). While we do not get direct supervision on preferences, we do get supervision on the choices users make (see Fig. 2). Using the choice model we assume in Sec. 3, we map the belief over preferences, together with the new slate st, to a distribution over items or choices bπ0:T (xt). At training time, we use the data Dj for each user and for each time step t to train the model via supervised learning, with a loss defined on the choice distribution outputted by the model, and the actual choice xt the user made. Such a model learns to perform approximate inference over ut in the underlying NHMM (see Appendix A).
Then at inference time, to predict future preferences for a user j under a new policy π′, we sample simulated user preference trajectories from the model to approximate Monte Carlo estimation in the NHMM. We use the data Dj as input to the model to obtain a belief over the next-timestep preferences uT+1. We then use such belief and the slate at time T + 1 from policy π′, sπ ′ T+1 = π′(s0:T , x0:T ), to simulate the user’s choice xπ ′
T+1. Treating this extra (simulated) step of interaction history for the user (the slate sπ ′
T+1 and choice x π′
T+1) as part of the observed history so far, we repeat this process to obtain preference estimates under π′ for future timesteps. By simulating multiple future trajectories, we obtain beliefs over the expected preferences a user (or a cohort of users) would have at any future timestep. See Algo. 1 for the exact procedure.
While the above method requires the learned future-preference predictor model to generalize to being able to incorporate histories collected under different RS policies than the ones in the training
data, there is still hope in that real-world datasets of user interaction were been collected under many different deployed policies. This suggests that as long as the slates chosen by π′ are not too dissimilar from π (a mixture policy that comprises all the policies used to gather the data), the network should be able to generalize to predict preference evolutions induced by π′ too.

heading: 4.2 COUNTERFACTUAL PREFERENCES ESTIMATION
text: While the methodology described in Sec. 4.1 is sufficient to estimate the expected preferences and behaviors a user will have in the future, it cannot be readily used to estimate counterfactual preferences and behaviors: given a set of past interactions Dj collected under policy π, we cannot predict what the preferences would have been for this user if an alternate policy π′ had been used from the first timestep of interaction instead of π – we denote these counterfactual preferences as uπ ′
0:T . On a high-level, to perform the counterfactual estimation task we want to extract as much information as possible about the user’s initial internal state from the historical interactions Dj we have available for them: even though such interactions were collected with a different policy π, they will still contain information about the user’s initial state (including their initial preferences before interacting with the RS). Then, based on this belief about the user’s initial internal state, we want to obtain a user-specific estimate of the effect that another policy π′ would have had on their preferences.
Estimation under known internal state dynamics. Under oracle access to the internal state dynamics, we can first obtain the belief over the initial state of a given user P(z0|sπ0:T , xπ0:T ) via NHMM smoothing – which is a trivial extension of HMM smoothing (Russell & Norvig, 2002) – and then roll out the human model forward dynamics with the fixed policy π′ instead of π: P(zπ′t |sπ0:T , xπ0:T ) = ∫ z0 P(zπ′t |z0)P(z0|sπ0:T , xπ0:T ) – equivalently to Sec. 4.1 when there is no past context but a specific prior for the internal state at t = 0.
Estimation under unknown internal state dynamics. Without oracle access to the internal state dynamics, again we try to learn to perform this NHMM counterfactual task approximately. One challenge in obtaining supervision for the task is that in our dataset of interactions we never get to see true counterfactuals. We get around this by decomposing the task into two parts, for which we use two separate models: (1) estimating the initial internal state for the user (based on the interaction data xπ0:T , s π 0:T under π which we have available), i.e. train a predictor which approximates the NHMM smoothing distribution P (u0|xπ0:T , sπ0:T ) which we will denote here as bπ0:T (u0) (the belief over initial preferences); and then (2) estimating uπ ′
0:T conditional on the belief of the initial preferences.
The models for the two tasks are trained with the same method used for predicting future preference estimation (Fig. 2) but with different inputs and supervision signal (Fig. 4). For task (1), the network is trained to predict the initial (instead of future) preferences of the user based on later context xπ1:T , s π 1:T . While the network predicts P (u0|xπ1:T , sπ1:T ), we can recover the correct smoothing estimate as P (u0|xπ0:T , sπ0:T ) ∝ P (xπ0 |sπ0 , u0)P (u0|xπ1:T , sπ1:T ) (see Appendix A). The network for task (2) is obtained by changing the prediction network to also condition on the recovered initial preferences (Fig. 2-4), enabling us to make predictions of the form P ( uπ ′
k+1|bπ0:T (u0), xπ ′ 0:k, s π′ 0:k
) . For training, we first recover initial preference beliefs for each user j
in the data Dj with network for task (1) (which we assume has already been trained). We then train the network for (2) to reconstruct the user j’s actual interactions (under π) simply based on this initial preference estimate (again, similarly to the case in Sec. 4.1 but by additionally conditioning on the initial preference belief). This teaches the network to leverage the information contained user’s initial preferences estimate to better estimate their preferences and behaviors based on what their interactions have been so far. In practice, we train these models using a transformer architecture similar to Sun et al. (2019), and we detail in Appendix A why this is a good fit for our tasks.
At inference time, we recover an estimate of initial preferences based on interaction data (sπ ′ 0:T , x π′
0:T ) of a new user with a new recommender policy π′ with model (1), and then estimate the preferences such user would have had under a safe policy πsafe with model (2). Such counterfactual preference estimate is obtained with Monte Carlo simulations similarly to Sec. 4.1 with the difference that the network is also conditioned on the initial preferences estimate. See Algo. 2 for the full algorithm and Appendix A for why this approximates the NHMM task.
We now have a way to estimate – for a new policy π′ – what we would expect its counterfactual impact would have been relative to having deployed π (from which we have observational data; note that similarly to future preferences estimation, this procedure too can suffer if π′ induces a strong distribution shift relative to the training data). We next discuss potential quantitative evaluation metrics defined on these estimates.

heading: 5 QUANTIFYING UNWANTED SHIFTS AND OPTIMIZING TO AVOID THEM
text: Given these estimates of preference shifts, it would be useful to have quantitative metrics for whether they are undesirable – as this would enable automatically evaluating policies and even actively optimizing to avoid such shifts. To obviate the difficulty of explicitly defining unwanted shifts, we limit ourselves to defining shifts that we trust (“safe shifts“ usafe0:T ), and flag preference shifts that largely differ from these safe shifts as potentially unwanted. This is similar to defining a region of the space which we trust in a risk-averse manner. The underlying philosophy is that “we don’t know what good shifts are, but as long as you stay close to [these shifts], things won’t go too poorly“. Therefore, we need both a notion of shifts we trust (“safe shifts“), and a notion of closeness (or distance) between induced preference shifts.
Notation. We denote as r̂t(usafet , π) the engagement at time t for the content that is chosen under the policy π – if it were evaluated under the safe-shift preferences usafet – i.e. r̂t(u safe t , π) = (x π t ) Tusafet .
Distance between shifts. We choose a metric between shifts such that engagement for the items xπ0:T chosen under policy π is also high under the preferences one would have had under safe shifts usafe0:T . This is operationalized as D(u π 0:T , u safe 0:T ) = ∑ t E [ (xπt ) Tuπt − (xπt )Tusafet ] = ∑ t E [ r̂t(u π t )− r̂t(u safe t , π) ] as our notion of distance between π-induced shifts and safe shifts usafe0:T .
Safe shifts: u0. As a first (very crude) proposal for safe shifts, we can consider no shifts at all: any deviation from the initial preferences u0 can be flagged as potentially problematic withD(uπ0:T , u0).
Safe shifts: natural preference shifts (NPS). Not all preference shifts are unwanted: people routinely change their preferences and behavior “naturally“. But what does “naturally“ even mean? We propose an idealized notion of natural shifts, by asking how preferences would evolve if the user were ”omniscient”, i.e. would have access to all content directly and had the ability to process it, unhindered by a small and biased slate offered by an RS. Intuitively, such a user’s preferences would still shift over time. Unfortunately this is impractical, as we can never get data from such hypothetical users that can attend to all content when choosing what to consume. As an approximation, we use random slates – these slates are still small, but they sampled randomly from the content, eliminating the bias of any RS policy (which in turn can change the user’s belief about the distribution of available content). We therefore operationalize “natural preferences“ uπrndt as the preferences which the user would have interacting with a random RS, and we use D(uπ0:T , u πrnd 0:T ) as the metric.
Penalized training. By training the models described in Sec. 4, one essentially obtains a human model which can be used to simulate human choices (and preferences). Similarly to previous work, we can use this human model as a simulator for RL training of recommender systems (Chen et al., 2019; Zhao et al., 2019; Bai et al., 2020). The metrics defined above give us a way to augment the reward function to penalize the policy for causing any shifts that we have not explicitly identified as “safe“ – in what can be considered a ”risk-averse” design choice. We incorporate these metrics in the training of a new RS policy π by adding the two distance metrics from above to the basic objective ∑T t E [ r̂t(u π t ) ]
of maximizing long-term engagement, leading to the updated objective∑T t E [ r̂t(u π t )+ν ′ 1 r̂t(u0, π)+ν ′ 2 r̂t(u πrnd t , π) ] where ν′1, ν ′ 2 are hyperparameters – see Appendix B.

heading: 6 EXPERIMENTAL SETUP
text: Why simulation? To test our method, we need to evaluate both RL policies that interact with users (rendering static datasets of user interaction unsuitable), as well as the metrics themselves, which are defined based on internal preferences (for which we never get ground truth in real interaction). We thus create a testing environment in which we can emulate user behavior and have access to their ground truth preferences. Like previous work (Chaney et al., 2018; Bountouridis et al., 2019; Jiang et al., 2019; Mansoury et al., 2020; Yao et al., 2021), we simulate both a recommendation environment and human behavior. However, unlike such approaches, we use simulated human behavior for testing purposes only: our human model is learned exclusively from data that would be observed in a real-world RS (slates and choices), i.e. that of users (in our case the simulated users) interacting with previous RS policies – meaning our approach could be applied to real user data of this form. A fundamental advantage of testing our methods in a simulated environment is that we can actually evaluate how well our model is able to recover the preferences of our ”ground truth” users, giving us insights about how our methods could perform with real users.
Ground truth human dynamics. See Fig. 9 & 5 for a summary of our environment setup and the ground truth human dynamics we use for testing our method. As mentioned in Sec. 3, we use the conditional logit model as choice model – in our setup, this reduces to P (xt = x|st, ut) ∝ P (st = x)e βcx Tut , with an additional term P (st = x) which takes into account how prevalent each item is in the slate (see Appendix C.1) – and we assume this to be also known by our method. We adapt Bernheim et al. (2021) to be our ground truth human preference dynamics. On a highlevel, at each timestep users choose their next-timestep preferences update their preference to more “convenient“ ones – trading off between choosing preferences that they expect will lead them to higher engagement and maintaining engagement under current preferences.
For ease of interpretation, we only consider a cohort of the RS’s users whose initial user preferences form a normal distribution around preference u = 130◦. To showcase preference-manipulation incentives to make users more predictable, we make the choice-stochasticity temperature βc a function of part of preference space one is in, with local optima βc(80◦) = 1 and βc(270◦) = 4 (see Appendix C.1 for more details). As users obtain higher engagement value when they act less stochastically, these portions of preference space form attractor points as can be seen in all policies in Fig. 1. While preferences naturally tend to shift towards one of these modes, some RS policies drive preferences to the other mode. As the engagement does not correspond to actual value, converging to the higher local optimum of engagement (u = 270◦ instead of u = 80◦) is not necessarily desirable.
Training human models and penalized RS policies. For training our human models, we use a BERT model (Sun et al., 2019), and only assume access to a dataset of historical interactions s0:T , x0:T . We train myopic and RL RS policies π′ using PPO (Schulman et al., 2017; Liang et al., 2018) and restrict the action space to 6 possible slate distributions for ease of training. For penalized training we give r̂t(uπt ), r̂t(u0, π ′), r̂t(u πrnd t , π ′) equal weight. See Appendix C for details.

heading: 7 RESULTS
text: Recovering preferences. Firstly, we confirm that our preference estimation models are able to recover very similar estimates to the equivalent NHMM inference tasks (which, unlike our method, needs access to the ground truth user preference dynamics) – see Appendix D for details, and Fig. 7 for an example of the estimated preferences relative to ground truth ones. This suggests that – under a correct choice model assumption and sufficient similarity between the RS policies represented in the historical datasets, and the ones we are interested in – our method is able to implicitly learn the preference dynamics of a user.
Hypotheses about metrics. We now turn to our hypotheses regarding metrics. We hypothesize that our metrics are able to: (H1) identify whether policies will induce unwanted preference shifts, and (H2) incentivize better behavior when added as a penalty during training.
Learned human model confound. With real users, to validate H1 one would have to rely on approximate computation of the metrics (estimated evaluation), as computing the metrics would require using our learned user models of preference dynamics. Additionally, to validate H2, one would likely train with simulated interactions from the learned user model (training in simulation) – as explained in Sec. 5. The quality of the learned user model would thus be a confound for testing the hypotheses, i.e. the quality of the form of the metrics themselves.
H1 under oracle dynamics access. In order to deconfound our experiments from the errors in our estimated human dynamics, we first test these hypotheses assuming oracle access to users – meaning that the policy would be deployed to them – and their dynamics. We first hypothesize that (H1.1) by computing the metrics exactly using the preference estimates obtained through NHMM inference (oracle evaluation), they are able to flag potentially unwanted preference shifts. We find this to be the case by comparing the oracle evaluation metric values (left of Table 1, ”unpenalized”) and the actual preference shifts induced by the various RSs we consider (Fig. 1): while unpenalized RL performs better (7.49 vs 5.71) than myopia for engagement r̂t(uπ ′
t ), it performs worse with respect to our safe shift metrics. This matches Fig. 1, where RL has more undesired effects.
H2 under oracle dynamics access. We additionally hypothesize that (H2.1) such metrics (still computed exactly, in oracle evaluation) can be used for training penalized LTV systems which avoid the most extreme unwanted preference shifts (and for this training, we will allow ourselves on-policy human interaction data with the ground truth users, as if the RL happened directly in the real world – we call this oracle training. For the penalized RL RS, the cumulative metric value (“Sum“ in Table 1) increases substantially, although it is at the slight expense of instantaneous engagement (Table 1). Qualitatively, we see that the induced preference shifts caused by the RL system seem closer to “safe shifts“ (Fig. 6), supporting H1.1 in that high metric values qualitatively match shifts that seem more desirable. Overall, we see that with oracle access, the metrics capture
what we see as qualitatively undesired shifts, and that optimizing for them directly (penalized RL) produces policies with slightly lower engagement, but drastically better at avoiding such shifts.
H1 and H2 with learned user dynamics. Our findings support our hypotheses under oracle access. However, in real life we will not have access to the underlying dynamics model, or even the ability to interact on-policy with users as we train RL policies, due to the high cost and risk of negative side effects for collecting data with unsafe policies. Therefore, we wish to show that even with estimated metrics and simulated interactions (based on the learned models), (H1.2) estimated evaluation and (H2.2) training in simulation (described above) are still able to respectively flag unwanted shifts and penalize manipulative RL behaviors. Table 2 shows that – although the estimated metrics can differ from the ground truth ones somewhat substantially (see Oracle Eval. at the top vs Estimated Eval. at the bottom) – importantly the relative ordering of the policies ranked by our estimated values stay the same: the penalized RL policy trained in simulation actually has (under oracle evaluation) higher cumulative reward than the unpendalized one, and the estimated evaluation keeps that ranking (even though is more optimistic about the unpenalized reward).
Table 1: Results under oracle training and evaluation. Both here an in Table 2 we report the cumulative expected engagement under a variety of different preferences (as explained in Sec. 5). The values are averaged across 1000 trajectories (standard errors are all < 0.1). By looking at
the safe shift metrics r̂(u0) and r̂(urndt ), we see that penalized systems stay significantly closer to safe shifts than unpenalized ones.
Oracle Training Unpenalized Penalized Myopic RL Myopic RL
O ra
cl e
E va
l r̂t(uπ ′
t ) 5.71 7.49 6.20 5.28 r̂t(u0, π
′) 1.99 -0.08 3.61 6.21 r̂t(u πrnd t , π
′) 2.01 -1.09 3.10 4.57 Sum 9.69 6.33 12.90 16.05

heading: 8 DISCUSSION
text: Summary. In conclusion, our contributions are 1) proposing a methodology to estimate the preference shifts which would be induced by recommender system policies before deployment, and 2) providing a formalism for defining distance metrics from safe preference shifts, which can be used both for evaluation and to train RSs which penalize unwanted shifts. As dynamics of preference are learned (rather than handcrafted), our method could be applied to real user data and has the potential for higher accuracy than handcrafted models due to massive amounts of behavior data. Further, while there is no ground truth for human preferences, verifying the model’s ability to anticipate behavior can give confidence in using it to evaluate and penalize undesired preference shifts.
Limitations and future work. We conducted our evaluation in simulation, which was necessary for testing the metrics themselves – for this we required ground truth access to user preferences. But since we used a specific dynamics model in our simulator, we cannot guarantee that our results translate when the method is applied to real users with real preference dynamics. We expect real users to have more complex dynamics, with preference changes mediated by beliefs outside of content distribution as in our simulator, such as beliefs about the world; while our method makes no assumptions about the structure of this internal space, it might require a lot more (and more diverse) data to capture these effects. Further, we also limited our evaluation to a setting explicitly designed such that an RL system would have incentives to manipulate preferences – not all real settings will necessarily have that property. Moreover, crucially our method requires assuming access to the user choice model: one could obviate this by predicting behavior and defining metrics on behavior, but this would mean losing the latent preference structure. Additionally, even what we call “preferences“ – de-noised behavior – is limiting as it doesn’t lend itself to capturing long-term preferences.

heading: A PREFERENCE ESTIMATION
text: Future preference estimation with Algorithm 1 and why it approximates NHMM prediction.
Algorithm 1 is used to obtain the belief over the preferences of a user would have at timestep H – assuming that the user interacted for T timesteps with a policy π (for which we have actual interaction data), and then interacted from timestep T to H with a policy π′. We refer to this belief as bπ0:T (uH) = P (u π′
H |sπ0:T , xπ0:T ). The first model inference pass in Algorithm 1 resulting in b0:T (uT+1) will be a distribution over uT+1 conditioned on sπ0:T , x π 0:T . A minimizer for this prediction problem would be for the network to represent the actual belief that would be obtained by performing the prediction step in the NHMM (described in Sec 4.1). This is because the expected cross-entropy between user choices and the induced choice distribution (induced by the preference belief prediction) would be minimized.
One limitation is that there are possibly multiple beliefs over preferences that induce the same exact distribution over choices: in that sense, the correct preference belief might be unidentifiable. Experimentally, we don’t find this to be a problem: the choice (see the ”Preference estimation model form” heading of this Appendix) of having the representation of the preference belief be a mixture of Von Mises distributions – which can be thought of as Normals over a circle – is already a good enough inductive bias to be able to recover good preference beliefs (see Appendix D).
In the NHMM, note that the unrolling the forward model (i.e. computing P(uπ′H |zπT+1)) could be computed exactly, or be performed with Monte Carlo estimation, by sampling many internal states zπT+1 from the forward prediction distribution, and then sampling internal state evolutions according to the dynamics. In our algorithm, by sampling choices and slates then conditioning on them, we are approximating the Monte Carlo estimation approach. When simulating each user choice, the model can be thought of as implicitly sampling a preference for this simulated user (from the preference belief) and then sampling a choice from the corresponding choice distribution. This means that each simulation rollout should be equivalent to ”sampling a user” (according to the distribution of users in the data) and then sampling their choices.
Algorithm 1: Predicting future user preferences at timestep H under policy π′
Inputs: past interactions xπ0:T , sπ0:T , a RS policy π′ with signature (s0:k, x0:k)→ sk+1, a future preference estimator P̂f with signature (s0:k, x0:k)→ (b0:k(uk+1), b0:k(xk+1)), a horizon H , number of Monte Carlo simulations N ; if x0:T = s0:T = ∅ then // if no past interaction data is given Sample slate s0 ∼ π′(∅, ∅) and imagine user choice x0 ∼ P̂ (x0|∅, ∅) ; // we now have past interactions with T = 0 for i = 0; i < N ; i++ do // sample N simulated trajectories
while k = T ; k < H; k ++ do b0:k(uk+1), b0:k(xk+1) = P̂f (s0:k, x0:k) ; // estimate pref. and choices sk+1 ∼ π′(x0:k, s0:k) ; // sample next timestep slate xk+1 ∼ b0:k(xk+1) ; // simulate a user choice Add xk+1 and sk+1 to the current simulated trajectory’s history;
Average b0:k−1(uk) and b0:k−1(xk) across the N futures (for each k, with T < k ≤ H); Result: Belief over future pref. bπ0:T (uk) and behaviors bπ0:T (xk) for each k s.t. T < k ≤ H .
Initial preference prediction model output correction
Below, we show that one can recover the smoothing estimate P (u0|x0:t, s0:t) from the predicted preferences P (u0|x1:t, s1:t) which will be a biased estimate of the initial preferences (as it does not incorporate the information from timestep t = 0) . Note that:
P (u0|x0:t, s0:t) = P (u0|x0, s0)P (x1:t, s1:t|u0) P (x0:t, s0:t) = P (u0|x0, s0) P (x0:t, s0:t) P (u0|x1:t, s1:t)P (x1:t, s1:t) P (u0)
(1)
= P (x0, s0|u0)P (u0|x1:t, s1:t)P (x1:t, s1:t)
P (x0:t, s0:t)P (x0, s0) ∝ P (x0, s0|u0)P (u0|x1:t, s1:t) (2)
The first equality is given by the definition of smoothing applied to t = 0 (Russell & Norvig, 2002). The second equality is obtained by using Bayes Rule on the backwards message P (x1:t, s1:t|u0), and the third is obtained by using Bayes Rule on P (u0|x0, s0). Finally, we can ignore P (x1:t, s1:t), P (x0:t, s0:t), and P (x0, s0) as they are constants.
Future preference estimation with Algorithm 2 and why it approximates NHMM prediction.
A similar argument to one in the above section can be made as to why such a initial preference estimation network would approximate the corresponding NHMM smoothing task.
However, for the second step of counterfactual preference estimation, one issue arises. Relative to having access to the full dynamics of the internal state, when performing approximate inference with our model we lose some information: we are only able to recover a belief over the initial preferences, whereas the NHMM with full dynamics access would be able to recover a belief over the full internal state of the user. This will reduce the accuracy of our counterfactual estimation, but is the best we can do without further assumptions.
Mathematically, we approximate the NHMM target distribution P (uπ ′
T |xπ0:t, sπ0:t) as:
P (uπ ′ T |xπ0:t, sπ0:t) ≈ ∫ u0 P (uπ ′ T |u0)P (u0|xπ0:t, sπ0:t) = P ( uπ ′ T |bπ0:t(u0) )
(3)
= ∑
xπ ′ 0:T−1,s π′ 0:T−1
P (xπ ′ 0:T , s π′ 0:T )P ( uπ ′ T |bπ0:t(u0), xπ ′ 0:T , s π′ 0:T ) (4)
where we the last expression is approximated with a Monte Carlo estimate detailed in Algorithm 21 (similarly to what was done in Algorithm 1). To do well at this second trajectory reconstruction task, the network necessarily needs to learn how to make best use of the belief over initial preferences, and implicitly learn their dynamics, as for the future preference estimation task.
Algorithm 2: Predicting counterfactual user preferences at timestep T under policy π′, given t timesteps of interaction data with π. Inputs: past interactions xπ0:t, sπ0:t, a RS policy π′ with signature (s0:k−1, x0:k−1)→ sk, an
initial preference estimator P̂i with signature (s0:k−1, x0:k−1)→ b0:k(u0), a conditioned future estimator P̂c with signature (s0:k−1, x0:k−1, b0:k(u0))→ (b0:k(uk+1), b0:k(xk+1)), a horizon T , a constant N ; bπ0:t(u0) = P̂i(s π 0:t, x π 0:t) ; // initial pref. belief given interactions with π P̂f = P̂c(b = b π 0:t(u0) ) ; // future pref predictor conditioned on init belief(
bπ0:k−1(u π′ k ), b π 0:k−1(x π′ k ) ) 0<k≤T =Algorithm 1 ( ∅, ∅, π′, P̂f , T,N ) Result: Distributions of counterfactual preferences and behaviors under policy π′
Preference estimation model form
We have three models for preference estimation: respectively initial, counterfactual, and future preference estimators P̂i, P̂c, P̂f . Any sequence model, such as RNNs or transformers, would be appropriate for these tasks that have variable number of inputs.
One detail of note that was omitted from Fig. 2 is that – to enable to represent multi-modal beliefs over preference space – we let the models’ output be parameters of multiple Von Mises distributions and additionally some weights w. The w weighted average of these distributions will form the predicted belief over ut.
In architecture, we use a similar form to that of BERT4Rec (Sun et al., 2019) for ease of performing inference on future or past preferences given contexts of interaction, as described in Fig. 8.
1This algorithm is not actually used in it’s pure form in the experiments. Our choice of ”safe policy” πrnd happens to choose constant slates (i.e. a uniform distribution no matter the history), so there is a shortcut to the procedure: by simply setting the inputted slated for counterfactual prediction to be uniforms, and predicting preferences without any user choices, one can train the model to directly output the belief over counterfactual preferences for any timestep, across the whole userbase.
Using transformers is not new to the context of recommendation for click prediction (Sun et al., 2019), but to our knowledge we are the first to leverage the flexibility of conditioning schemes enabled by the BERT architecture (Devlin et al., 2019; Joshi et al., 2020; Wang & Cho, 2019).
Specifically, we train 3 transformers (one for prediction, one for smoothing, and one for counterfactual estimation), although preliminary results show that one can train a single transformer to perform all three tasks without a significant loss in performance. All together, these 3 transformers constitute the “learned human model“ which we refer to throughout the text.

heading: B PENALIZED RL

heading: The underlying MDP
text: One could cast the recommendation problem as a POMDP (Lu & Yang, 2016; Mladenov et al., 2019) in which the state of the environment is hidden and contains the user’s internal state, which evolves over time. Equivalently, one can consider the belief-MDP induced by the recommender POMDP (Kaelbling et al., 1998), and approximate a solution to such belief-MDP via Deep-RL with a policy trained with observation histories as input (this is theoretically sufficient for the policy to recover a belief over the current hidden state and take the optimal action). The action space will be given by the space of possible slates that the RS can choose. The reward signal will be the expected reward for the current timestep E [ r̂t(u π t ) ]
(or with the extra terms for the proxies in the case of penalized training). The introduction of expectation can be thought of as expected SARSA (Sutton & Barto, 1998), as argued in (Ie et al., 2019).

heading: Penalized RL training
text: The full set of steps to run RL training are as follows: once the human models described in Sec. 4 are trained, one can use them to simulate user trajectories and compute penalty metrics for such trajectories (see Algorithm 3). One can then optimize the RL policy based on the on-policy simulated trajectory rollouts.
Reduction of distances to final penalized objective Note that the full penalized RL objective (∑T t E [ r̂t(u π t ) ]) − ν1D(uπ0:T , u0) − ν2D(uπ0:T , u πrnd 0:T )
for our choice of distance function D reduces to ∑T t E [ r̂t(u π t ) + ν ′ 1 r̂t(u0, π) + ν ′ 2 r̂t(u πrnd t , π) ] for some choice of ν′1, ν ′ 2 which can be treated as hyperparameters of how much we want to value the engagement under each safe policy preferences relative to the engagement under the main policy.

heading: C EXPERIMENT DETAILS

heading: C.1 GROUND TRUTH USERS
text: Reduction of logit model to our case.
Algorithm 3: Generating a trajectory for RL training and computing metrics
Inputs: Initial, counterfactual, and future preference estimators P̂i, P̂c, P̂f ; a policy π, a safe policy πsafe, a horizon H , a constant N . Sample slate sπ0 ∼ π(∅, ∅) and imagine user choice xπ0 ∼ P̂i(x0|∅, ∅); for t = 1; t ≤ H; t++ do
bπ0:t−1(u0) = P̂i(s π 0:t−1, x π 0:t−1) ; // current belief over initial preferences bπ0:t−1(ut), b π 0:t−1(xt) = P̂f (s π 0:t−1, x π 0:t−1) ; // belief over pref and choices E[bπ0:t−1(u πsafe t )] ∈ Algorithm 2 ( t, πsafe, P̂c, P̂i, x π 0:t−1, s π 0:t−1 ) ; // belief over counterfactual
preferences under safe policy for this user sπt ∼ π(xπ0:t−1, sπ0:t−1) ; // sample slate xπt ∼ bπ0:t−1(xt); // imagine a user choice Dt(u π t , u safe t ) = Euπt ,uπsafet ,xπt [ (xπt ) Tuπt − (xπt )Tu πsafe t ] ; // compute penalty metric(s) for
timestep rRLt = E[rt] +Dt(uπt , usafet )
Return: User-RS interactions and training rewards for the simulated trajectory;
In our experimental setup, the traditional conditional logit model P (xt = x|st, ut) = e βcx T ut∑ x∈s e βcxT ut doesn’t apply directly in this form, as we consider slates to be distributions rather than sets of discrete items. Intuitively, user’s choices should still depend on the slate: the proportion of a certain item in the current slate (one can think of this as the proportion of a certain item type), should influence the probability of the user of selecting that item (type). We operationalize this as P (xt = x|st, ut) ∝ P (st = x)e βcx Tut , with an additional term P (st = x) which takes into account the proportion of each item (type). Note that this also mathematically corresponds to the generalization of the traditional logit model: when the slate is discrete, P (st = x) will simply be an indicator for whether the item is in the slate, leading to the traditional logit model form.

heading: Feed belief update
text: Our ground truth user has a belief over future slates bHt (s). Users’ initial belief matches the content feature distribution itself bH0 (s) = D. After receiving a slate st, the user’s induces a belief bHt (s) ∝ s3t over the future slates in the user, i.e. the user will expect the next feeds to look like the most common items in the current feed, as a result of availability bias (MacLeod & Campbell, 1992).

heading: Lack of no-op choice.
text: While the assumption that user must pick item from the slate is unrealistic, this could be resolved by adding an extra no-op choice to every slate (Sunehag et al., 2015).

heading: Prefernce shift model.
text: Our preference shift model is inspired by Bernheim et al. (2021), but adapted to our experimental setup as described below. The choice of preferences is modulated by a “mindset flexibility“ parameter λ which captures how open they are to modifying their current preferences. Users assign value to the choice of next-timestep preferences ut+1 as: V ( ut, b H t (s), ut+1, λ ) = Ext+1∼bHt (s)[λr̂t+1(ut) + (1− λ)r̂t+1(ut+1]), where with r̂t+1(ut) indicates the engagement value obtained by the user under the choice xt+1 and preference ut. Users pick their next timestep preferences also according to the conditional logit model, but over their expected value of such preferences choices: P (ut+1|bHt (s), ut, λ) ∝ eβdV (ut,b H t (s),ut+1,λ). Intuitively, users update their preference to more “convenient“ ones – ones that they expect will lead them to higher engagement value. The main change we introduce from the original model is incorporating the belief over future feeds.
User’s preference-flexibility parameter is given by λ = 0.9, and their initial preferences are drawn from a normal distribution2 around preference u = 130◦ and standard deviation 20◦.
2Technically one should use Von Mises distributions – a distribution similar to normals, but for which the domain is a circle. As Von Mises distributions are not implemented in numpy (Harris et al., 2020), for simplicity we use clipped normal distributions (disregarding probability mass beyond 180◦ in either direction) in all places except for the the transformer output (which was implemented in PyTorch (Paszke et al., 2019), which has a Von Mises implementation)

heading: C.2 LEARNED HUMAN MODELS
text: For our learned human models, we use BERT transformers with 2 layers, 2 attention heads, 4 sets of Von Mises distribution parameters, a learning rate of 0.00003, batch size of 500, and 100 epochs. We train on the data described below.

heading: C.3 SIMULATED DATASET
text: See Fig. 9 for a summary of how content is generally instantiated in our setup.
We set the distribution of content in such a way that it forms a uniform distribution across features, that is D = Uniform(0◦, 360◦). We simulate historical user interaction data with a mixed policy π which is similar (but not equal to) a random policy in half the rollouts and for the other half is goal-directed:
• Half of the data is created with a RS policy which chooses an action uniformly among a set of possible slates (distributions over the feature space with means 0◦, 10◦, ..., 340◦, 350◦, and standard deviations equal to 30◦ or 60◦). Each of these slate types can be thought of as a slate which contain mostly one specific type of content.
• The other half of the data is created with a RS which chooses st = D = Uniform(0◦, 360◦) 80% of the time (i.e. the slate that would be chosen by the random RS πrnd), and chooses a random action from the same set of possible slates the remaining 20% of the time.
This is to simulate the setting in which the safe policy we are interested in (the random NPS policy) is similar to previously deployed ones that are represented in the data (although not the same, so the network still has to generalize across RS policies at test time).
Our dataset consists of 100k user trajectories (slates and observations), with length of each trajectory T = 10.

heading: C.4 RS TRAINING
text: For RL optimization, we use PPO (Schulman et al., 2017) trained with Rllib (Liang et al., 2018). The action space of the recommender system is given by distributions over feature space with means 0◦, 60◦, ..., 260◦, 320◦, and standard deviations equal to 60◦. As observations to the system, we provide the current slate, the previous user choice, and the current estimates from the HMM for smoothing, filtering, and natural preference shift counterfactual distributions, in order to increase training speed (note that these will not change the optimal policy). All policies are recurrent so they are able to reason about the history of interactions so far.
We use batch size 1200, minibatch size 600, 4 parallel workers, 0.005 learning rate, 50 gradient updates per minibatch per iteration, policy function clipping parameter of 0.5, value function clipping parameter of 50 and loss coefficient of 8, with an LSTM network with 64 cell size. Training runs in less than 30 minutes for each condition on a MacBook Pro 16” (2020).

heading: D RESULTS
text: Quality of preference estimation
We baseline the quality of our counterfactual estimates of user preferences and behaviors relative to the performance of NHMM with full access to the human dynamics – as that is the best performance we could possibly hope for. The metrics we use to evaluate the performance of the transformer relative to the HMM are the 1) relative prediction loss (RPL) as the ratio between the transformer loss and the HMM loss ( LTrLHMM ), and the 2) relative prediction accuracy (RPA), defined similarly as AccTr AccHMM . Given that NHMM is the gold standard of estimation (as we are doing exact inference), RPL should be larger than 1 and RPA should be lower than 1. The closer to 1 the numbers are, the better performing our approximate inference methods are. For all the models, we train on a 75k user interaction trajectories dataset obtained from the distribution described in C. For all the tasks below, we use the methodology developed in Sec. 4.1.
Future preference estimation We train a transformer model as in the left portion of Fig. 8, by predicting – for any given timestep t – the preferences ut (and thus also a distribution over item choices) based on slates s0, . . . , st−1 and the previous observations x0, . . . , xt−1. All other inputs are masked. We evaluate on a held out dataset of 25k user interaction trajectories, which are obtained from the same distribution as the training data. The preference RPL and RPA (averaged across all timesteps) are 1.08 and 0.88.
Initial preference estimation We train a transformer model as in the middle portion of Fig. 8, by predicting the preferences u0 (and thus also a distribution over item choices) based on slates s1, . . . , sT and the observations x1, . . . , xT . All other inputs are masked. We evaluate on the same held out dataset as the prediction case. The preference RPL for the initial timestep is 1.08, and the preference RPA is 0.81.
Counterfactual preference estimation We train a transformer model as in the right portion of Fig. 8, by predicting – for any given timestep t – the preferences ut (and thus also a distribution over item choices) based on slates s0, . . . , sT and the observations x1, . . . , xT – i.e. reconstructing the original trajectory from the smoothing estimate obtained from the network described above. We evaluate on a held out dataset of 25k user interaction trajectories obtained from the natural preference shift policy πrnd – showcasing the generalization performance of our method to different policies. The preference RPL (averaged across timesteps) is 1.16, and the preference RPA is 0.63 – showing that although the task is more challenging for the models, performance doesn’t degrade drastically relative to the NHMM. Fig. 7 shows this qualitatively: the counterfactual network is able to estimate what natural preference shifts would look like relatively well relative to the ground truth across the population.

Review summaries:
This paper studies an interesting problem of estimating preference shifts induced by recommender systems and proposes a distance metric for the shifted preferences based on the so-called "safe shifts". The proposed metric can help penalize undesired preference shifts, which can be easily used in training and evaluation. Experiments via the simulated recommendation environment and simulated human behavior verify the effectiveness of the proposed approach. 

This paper discusses an interesting but important problem in recommender systems, i.e., the users’ preference shift under the influence of recommender systems. Currently, both RL-based and myopic solutions cannot avoid influencing preferences in undesirable ways. Instead, authors provide a simulation framework that enables system designers to estimate the induced preference shifts an RS; evaluate its influence before deployment; and actively optimize to avoid such shifts. The basic idea is two-fold. The first design is to estimate user preference dynamics by training a user predictive model from historical user interaction data, which supports both future preference estimation and counterfactual estimation. The second one is to evaluate and optimize the current recommendation policy by measuring whether its influence is close to users’ natural preference shift.  By simulation study, authors show that recommender systems optimizing for staying in the above trust-region can avoid manipulative behaviors (e.g., changing preferences in ways that make users more predictable). 

The authors address the challenge of preference shifts, a potentially undesired effect of recommendation systems (RSs). A preference shift is a byproduct of any RL system, in which the systems drive users to places (i.e., in their particular preference representation) where they are easily satisfiable. This can happen, e.g., by positive feedback loops. The main challenge of tacking this undesired effect is that we lack the “natural preferences” or a robust way to predict preference shift given a recommendation policy. The authors suggest building a system, which is partially data-driven and partially based on behavioral assumptions to simulate recommendation policies and find safe shifts --- shifts ensuring that users’ preferences will not be influenced too much by the specific policy. They explain the ingredients of the system and experimentally test it.