Estimating and Penalizing Induced Preference Shifts in Recommender Systems

The actions that a recommender system ( RS ) takes – the content it exposes users to – influence the preferences users have over what content they want . Therefore , when an RS designer chooses which system to deploy , they are implicitly choosing how to shift or influence user preferences . Even more , if the RS is trained via long-horizon optimization ( e.g . reinforcement learning ) , it will have incentives to manipulate preferences , i.e to shift them so they are more easy to satisfy , and thus conducive to higher reward . While some work has argued for making systems myopic to avoid this issue , myopic systems can still influence preferences in undesirable ways . In this work , we argue that we need to enable system designers to estimate the shifts an RS would induce ; evaluate , before deployment , whether the shifts are undesirable ; and even actively optimize to avoid such shifts . These steps involve two challenging ingredients : estimation requires the ability to anticipate how hypothetical policies would influence user preferences if deployed – we do this by training a user predictive model that implicitly contains their preference dynamics from historical user interaction data ; evaluation and optimization additionally require metrics to assess whether such influences are manipulative or otherwise unwanted – we introduce the notion of “ safe shifts ” , that define a trust region within which behavior is believed to be safe . We show that recommender systems that optimize for staying in the trust region can avoid manipulative behaviors ( e.g. , changing preferences in ways that make users more predictable ) , while still generating engagement . 1 INTRODUCTION . A recommender system ( RS ) exposes users to feeds of items , and users choose what to click on based on their preferences . These preferences are non-stationary : they shift over time ( Koren , 2009 ; Rafailidis & Nanopoulos , 2016 ; Li et al. , 2014 ) . Importantly , these shifts are not independent from the RS ’ s actions : what content the RS exposes users to influences their preferences . Because of this influence , when a system designer chooses a specific recommender algorithm ( policy ) , they are implicitly choosing how to shift user preferences . A growing trend is to choose policies that stem out of long-term optimization of user engagement ( Afsar et al. , 2021 ) – typically via reinforcement learning ; we will refer to these as long-term value , or LTV , systems . However , these policies might lead to very undesirable shifts as a side-effect : certain preferences are easier to satisfy than others , leading to more potential for engagement – this could be because of availability of more content for some preferences compared to others , or because very strong preferences for a particular type of content lead to higher engagement than more neutral ones . The LTV system therefore has an incentive to drive user preferences over time into these areas of higher engagement potential . This can make such LTV systems a particularly poor choice for avoiding undesired shifts . While it has been proposed to stifle RSs ’ capabilities by making them myopic – to avoid the incentives for preference manipulation above ( Krueger et al. , 2020 ) – even myopic systems can influence preferences in systematically undesirable ways ( Jiang et al. , 2019 ; Mansoury et al. , 2020 ; Chaney et al. , 2018 ) . Fig . 1 shows a simple example , plotting user preferences in a 1D space over time ( in a simulated environment ) . On the right , we see how interacting with the RL policy drives users strongly away from their initial preferences to a specific small region . The myopic policy ( center ) , shows the same type of effect , but much less pronounced , where after 10 time steps some users do reach other preferences as well . The reality is that any policy will have influence on user preferences . In this work , we argue that we need to explicitly account for this influence when choosing which policy to deploy . This requires tackling two critical problems . First , we need to anticipate what preference shifts an RS policy will cause before it is deployed . This alone would enable system designers to qualitatively assess whether a policy induces shifts that seem manipulative or otherwise unwanted . We study how to do this from historical user interaction data of previous RS policies , and show how to train a predictive model that implicitly captures preference dynamics . The second critical problem is having metrics for quantitatively evaluating whether an evolution of preferences is actually unwanted : a computable metric not only simplifies evaluation , but also enables RSs to actively optimize against unwanted shifts . Unfortunately , defining what “ undesirable “ shifts are is a complex philosophical and ethical challenge ( noa , 2018 ; Paul , 2014 ) . To side-step it , we propose a conservative approach based on the notion of ” safe shifts ” : instead of defining what is undersirable , we non-exhaustively define certain shifts that we trust not to be problematic , and measure the extent to which a policy deviates from them . For instance , a candidate for safe shifts that we introduce is how user preferences would naturally shift if the user were ” omniscient ” , i.e . if they could see and attend to the true distribution of available content without any bias from the RS selecting specific slates to show them . Fig . 1 ( left ) shows this natural preference evolution for our running example , and how user preferences stay somewhat diffuse but drift towards the opposite mode that the RL and myopic policies push them to . Note that while this metric can effectively penalize undersired shifts , it comes at a cost : natural shifts , and in fact any lists of safe shifts that we define , are unlikely to be exhaustive , which means the approach will conservatively penalize policies that might in reality be safe . To demonstrate both the estimation of preference shifts and their evaluation , we set up a testing environment in which we emulate ground truth user behavior by drawing from a model of preference shift from prior work ( Bernheim et al. , 2021 ) . We first show qualitatively that in this environment , RL and even myopic RS lead to undesired shifts . We then find that our preference shift estimation model , if trained based on historical user interaction with different policies , can anticipate these shifts ( see Fig . 7 ) . Further , we find that our evaluation metric correctly flags which policies will produce undesired shifts , and evaluates the RL policy from Fig . 1 as 3̃5 % worse than the myopic one , which is in turn 4̃0 % worse than a policy which avoids shifting preference to the ” unnatural ” mode from the figure . Our results also suggests that evaluating our metric using the trained estimation model correlates to using ground truth preference dynamics , and that optimizing for safe shifts does lead to higher scoring ( more safe ) policies . Although this just scratches the surface of finding the right metrics for unwanted preference shifts and evaluating them , our results already have implications for the development of recommender systems : in order to ethically use systems at scale , we e active steps to measure and penalize how such systems shift users ’ internal states . In turn , we offer hope that this is possible – at least for preference shifts – by learning from user interaction data , and put forward a framework for specifying “ safe shifts “ for detecting and controlling such incentives . 2 RELATED WORK . RS effects on users ’ internal states . A variety of previous work considers how RSs can affect users : influencing user ’ s preferences for e-commerce purposes ( Häubl & Murray , 2003 ; Cosley et al. , 2003 ; Gretzel & Fesenmaier , 2006 ) , altering people ’ s moods for psychology research ( Kramer et al. , 2014 ) , changing populations ’ opinions or behaviors ( Matz et al. , 2017 ) , exacerbate ( Hasan et al. , 2018 ) cases of addiction to social media ( Andreassen , 2015 ) , or increase polarization ( Stray , 2021 ) . There have been three main types of approaches to quantitatively estimating effects of RSs ’ policies on users : 1 ) analyzing static datasets of interactions directly ( Nguyen et al. , 2014 ; Ribeiro et al. , 2019 ; Juneja & Mitra , 2021 ; Li et al. , 2014 ) , 2 ) proposing hand-crafted models of user dynamics and using them to simulate interactions with RSs ( Chaney et al. , 2018 ; Bountouridis et al. , 2019 ; Jiang et al. , 2019 ; Mansoury et al. , 2020 ; Yao et al. , 2021 ) , or 3 ) using access to real users and estimating effects through direct interventions ( Holtz et al. , 2020 ; Matz et al. , 2017 ) . Our approach is most similar to 2 ) , but we propose implicitly learning user dynamics instead of handcrafting our own model . Neural networks for recommendation and human modeling . While data-driven models of human behavior have been used widely in real-world RS as click predictors ( Zhang et al. , 2019 ; Covington et al. , 2016 ; Cheng et al. , 2016 ; Okura et al. , 2017 ; Mudigere et al. , 2021 ; Zhang et al. , 2014 ; Wu et al. , 2017 ) and for simulating human behavior in the context RL RSs ’ training ( Chen et al. , 2019 ; Zhao et al. , 2019 ; Bai et al. , 2020 ) , to our knowledge they have not been previously used for simulating and quantifying the effect of hypothetical recommender systems on users . We emphasize how human models can also be used as auditing mechanisms by anticipating RSs ’ policies impact on users ’ behaviors and , as our method shows , even preferences . RL for RS . Using RL to train LTV RSs has recently seen a dramatic increase of interest ( Afsar et al. , 2021 ) , with some notable examples being proposed by YouTube and Facebook ( Ie et al. , 2019 ; Chen et al. , 2020 ; Gauci et al. , 2019 ) – which reported significant engagement increases with real users . Singh et al . in particular also takes a penalized RL approach . Side effects and safe shifts . Our work starts from a similar question to that of the side effects literature ( Krakovna et al . ; Kumar et al. , 2020 ) , but applied to preference change : given that the reward function will be misspecified , how do we prevent undesired preference-shift side effects ? The choice of safe shifts corresponds to the choice of baseline in this literature . 3 PRELIMINARIES . Setup . We model users as having time-indexed preferences ut ∈ Rd , which assign a scalar value to every possible item of content x ∈ Rd . and the user engagement derived from the consumption of item xt under ut at time t is modeled as being given by r̂t ( ut ) = uTt xt . These preferences are part of the user ’ s internal state zt , along with other variables , which together with preferences comprise a sufficient statistic for their long-term behavior , but which our method will not explicitly model . At every time step , the user sees a slate st produced by that RS policy π , and chooses an item xt . The policy π maps history of slates and choices so far , s0 : t , x0 : t , to the new slate st+1 . Upon making a choice , the user ’ s internal state updates to zt+1 . User choice model . We are interested in inferring and predicting preferences , which we do not directly observe . As such , we make an assumption about how user behavior – their choice xt – relates to their current preferences ut . Following prior work ( Ie et al. , 2019 ; Chen et al. , 2019 ) , we assume that users choose items in proportion to ( the exponent of ) the engagement under their current preferences , i.e . that P ( xt = x|st , ut ) is given by the conditional logit model ( detailed in Sec . 6 ) . Casting this as an NHMM . The dynamics of user internal state – which we don ’ t assume to be known – depend on the history so far because the policy π uses history . This makes our setup a nonhomogeneous HMM ( Hughes et al. , 1999 ; Rabiner ) , with hidden state zt and dynamics that timedependent : P ( zt+1|zt , π ( s0 : t , x0 : t ) ) . In our method will not commit to an explicit representation of zt , and will only implicitly learn these dynamics . For our experiments however , we will build a testing environment that does commit to a representation of zt via preferences and user beliefs , and adopts a ” ground truth ” dynamics of this internal state based on prior literature .

Summary: The authors address the challenge of preference shifts, a potentially undesired effect of recommendation systems (RSs). A preference shift is a byproduct of any RL system, in which the systems drive users to places (i.e., in their particular preference representation) where they are easily satisfiable. This can happen, e.g., by positive feedback loops. The main challenge of tacking this undesired effect is that we lack the “natural preferences” or a robust way to predict preference shift given a recommendation policy. The authors suggest building a system, which is partially data-driven and partially based on behavioral assumptions to simulate recommendation policies and find safe shifts --- shifts ensuring that users’ preferences will not be influenced too much by the specific policy. They explain the ingredients of the system and experimentally test it.