Hidden Incentives for Auto-Induced Distributional Shift

1 INTRODUCTION . Consider a content recommendation system whose performance is measured by accuracy of predicting what users will click . This system can achieve better performance by either 1 ) making better predictions , or 2 ) changing the distribution of users such that predictions are easier to make . We propose the term auto-induced distributional shift ( ADS ) to describe this latter kind of distributional shift , caused by the algorithm ’ s own predictions or behaviour ( Figure 1 ) . ADS are not inherently bad , and are sometimes even desirable . But they can cause problems if they occur unexpectedly . It is typical in machine learning ( ML ) to assume ( e.g . via the i.i.d . assumption ) that ( 2 ) will not happen . However , given the increasing real-world use of ML algorithms , we believe it is important to model and experimentally observe what happens when assumptions like this are violated . This is the motivation of our work . In many cases , including news recommendation , we would consider ( 2 ) a form of cheating—the algorithm changed the task rather than solving it as intended . We care which means the algorithm used to solve the problem ( e.g . ( 1 ) and/or ( 2 ) ) , but we only told it about the ends , so it didn ’ t know not to ’ cheat ’ . This is an example of a specification problem ( Leike et al. , 2017 ; Ortega et al. , 2018 ) : a problem which arises from a discrepancy between the performance metric ( maximize accuracy ) and “ what we really meant ” : in this case , to maximize accuracy via ( 1 ) . Ideally , we ’ d like to quantify the desirability of all possible means , e.g . assign appropriate rewards to all potential strategies and “ side-effects ” , but this is intractable for real-world settings . Using human feedback to learn reward functions which account for such impacts is a promising approach to specifying desired behavior ( Leike et al. , 2018 ; Christiano et al. , 2017 ) . But the same issue can arise whenever human feedback is used in training : one means of improving performance could be to alter human preferences , making them easier to satisfy . Thus in this work , we pursue a complementary approach : managing learners ’ incentives . A learner has an incentive to behave in a certain way when doing so can increase performance ( e.g . accuracy or reward ) . Informally , we say an incentive is hidden when the learner behaves as if it were not present . But we note that changes to the learning algorithm or training regime could cause previously hidden incentives to be revealed , resulting in unexpected and potentially undesirable behaviour . Managing incentives ( e.g . controlling which incentives are hidden/ revealed ) can allow algorithm designers to disincentivize broad classes of strategies ( such as any that rely on manipulating human preferences ) without knowing their exact instantiation.1 The goal of our work is to provide insights and practical tools for understanding and managing incentives , specifically hidden incentives for auto-induced distributional shift : HI-ADS . To study which conditions cause HI-ADS to be revealed , we present unit tests for detecting HI-ADS in supervised learning ( SL ) and reinforcement learning ( RL ) . We also create an environment that models ADS in news recommendation , illustrating possible effects of revealing HI-ADS in this setting . The unit tests both have two means by which the learner can improve performance : one which creates ADS and one which does not . The intended method of improving performance is one that does not induce ADS ; the other is ’ hidden ’ and we want it to remain hidden . A learner `` fails '' the unit test if it nonetheless pursues the incentive to increase performance via ADS . In both the RL and SL unit tests , we find that introducing an outer-loop of meta-learning ( e.g . Population-Based Training ( PBT ) Jaderberg et al . ( 2017 ) ) can lead to high levels of failure . Similarly , recommender systems trained with PBT induce larger drifts in user base and user interests . These results suggest that failure of our unit tests indicates that an algorithm is prone to revealing HI-ADS in other settings . Finally , we propose and test a mitigation strategy we call context swapping . The strategy consists of rotating learners through different environments throughout learning , so that they can ’ t see the results or correlations of their actions in one environment over longer time horizons . This effectively mitigates HI-ADS in our unit test environments , but did not work well in content recommendation experiments . 2 BACKGROUND . 2.1 META-LEARNING AND POPULATION BASED TRAINING . Meta-learning is the use of machine learning techniques to learn machine learning algorithms . This involves instantiating multiple learning scenarios which run in an inner loop ( IL ) , while an outer loop ( OL ) uses the outcomes of the inner loop ( s ) as data-points from which to learn which learning algorithms are most effective ( Metz et al. , 2019 ) . The number of IL steps per OL step is called the interval . Many recent works focus on multi-task meta-learning , where the OL seeks to find learning rules that generalize to unseen tasks by training the IL on a distribution of tasks ( Finn et al. , 2017 ) . Single-task meta-learning includes learning an optimizer for a single task ( Gong et al. , 2018 ) , and adaptive methods for selecting models ( Kalousis , 2000 ) or setting hyperparameters ( Snoek et al. , 2012 ) . For simplicity in this initial study we focus on single-task meta-learning . Population-based training ( PBT ; Jaderberg et al. , 2017 ) is a meta-learning algorithm that trains multiple learners L1 , ... , Ln in parallel , after each interval ( T steps of IL ) applying an evolutionary OL step which consists of : ( 1 ) Evaluate the performance of each learner , ( 2 ) Replace both parameters and hyperparameters of 20 % lowest-performing learners with copies of those from the 20 % high- 1Note removing or hiding an incentive for a behavior is different from prohibiting that behavior , which may still occur incidentally . In particular , not having a ( revealed ) incentive for behaviors that change a human ’ s preferences , is not the same as having a ( revealed ) incentive for behaviors that preserve a human ’ s preferences . The first is often preferable ; we don ’ t want to prevent changes in human preferences that occur “ naturally ” , e.g . as a result of good arguments or evidence . performing learners ( EXPLOIT ) . ( 3 ) Randomly perturb the hyperparameters ( but not the parameters ) of all learners ( EXPLORE ) . 2.2 DISTRIBUTIONAL SHIFT AND CONTENT RECOMMENDATION . In general , distributional shift refers to change of the data distribution over time . In supervised learning with data x and labels y , this can be more specifically described as dataset shift : change in the joint distribution of P ( x , y ) between the training and test sets ( Moreno-Torres et al. , 2012 ; Quionero-Candela et al. , 2009 ) . As identified by Moreno-Torres et al . ( 2012 ) , two common kinds of shift are : ( 1 ) Covariate shift : changing P ( x ) . In the example of content recommendation , this corresponds to changing the user base of the recommendation system . For instance , a media outlet which publishes inflammatory content may appeal to users with extreme views while alienating more moderate users . This self-selection effect ( Kayhan , 2015 ) may appear to a recommendation system as an increase in performance , leading to a feedback effect , as previously noted by Shah et al . ( 2018 ) . This type of feedback effect has been identified as contributing to filter bubbles and radicalization ( Pariser , 2011 ; Kayhan , 2015 ) . ( 2 ) Concept shift : changing P ( y|x ) . In the example of content recommendation , this corresponds to changing a given user ’ s interest in different kinds of content . For example , exposure to a fake news story has been shown to increase the perceived accuracy of ( and thus presumably future interest in ) the content , an example of the illusory truth effect ( Pennycook et al. , 2019 ) . For further details on such effects in content recommendation , see Appendix 8 . 3 AUTO-INDUCED DISTRIBUTION SHIFT ( ADS ) . Auto-induced distribution shift ( ADS ) is distributional shift caused by an algorithm ’ s behaviour . This is in contrast to distributional shift which would happen even if the learner were not present - e.g . for a crash prediction algorithm trained on data from the summer , encountering snowy roads is an example of distributional shift , but not auto-induced distributional shift ( ADS ) . a1 a2 s1 s2 r1 r2 ( a ) RL : Incentives for ADS are present ; pursuing them is desirable a1 a2 s1 s2 r1 r2 ( b ) Myopic RL : Incentives for ADS are present ; pursuing them is undesirable ŷ1 ŷ2 x1 x2 ` 1 ` 2 ( c ) SL with ADS : Incentives for ADS are present ; pursuing them is undesirable ŷ1 ŷ2 x1 x2 ` 1 ` 2 ( d ) SL with i.i.d . data : Incentives for ADS are absent Figure 2 : The widely studied problems of reinforcement learning ( RL ) with state s , action a , reward r tuples , and i.i.d . supervised learning ( SL ) with inputs x , predictions ŷ and loss l ( a , d ) are free from incentive problems . We focus on cases where there are incentives present which the learner is not meant to pursue ( b , c ) . Lines show paths of influence . The learner may have incentives to influence any nodes descending from its action , A , or prediction , ŷ . Which incentives are undesirable ( orange ) or desirable ( cyan ) for the learner to pursue is context-dependent . We emphasize that ADS are not inherently bad or good ; often ADS can even be desirable : consider an algorithm meant to alert drivers of imminent collisions . If it works well , such a system will help drivers avoid crashing , thus making self-refuting predictions which result in ADS . What separates desirable and undesirable ADS ? The collision-alert system alters its data distribution in a way that is aligned with the goal of fewer collisions , whereas the news manipulation results in changes that are misaligned with the goal of better predicting existing users ’ interests ( Leike et al. , 2018 ) . In reinforcement learning ( RL ) , ADS are typically encouraged as a means to increase performance . On the other hand , in supervised learning ( SL ) , the i.i.d . assumption precludes ADS in theory . In practice , however , the possibility of using ADS to increase performance ( and thus an incentive to do so ) often remains . For instance , this occurs in online learning . In our experiments , we explicitly model such situations where i.i.d . assumptions are violated : We study the behavior of SL and myopic RL algorithms , in environments designed to include incentives for ADS , in order to understand when incentives are effectively hidden . Fig . 2 contrasts these settings with typical RL and SL .

Summary: This paper discusses a phenomenon where machine-learned models may influence user behaviors in future iterations, creating self-selection effects such as filter bubbles or propagation of fake news. The paper calls these effects auto-induced distribution shift (ADS) and argues that a specific meta-learning algorithm PBT manipulates users instead of maximizing rewards. To illustrate the ideas, the paper introduces a few simulation environments where user behaviors may change when certain prediction events happen.