Hidden Incentives for Auto-Induced Distributional Shift

1 INTRODUCTION . Consider a content recommendation system whose performance is measured by accuracy of predicting what users will click . This system can achieve better performance by either 1 ) making better predictions , or 2 ) changing the distribution of users such that predictions are easier to make . We propose the term auto-induced distributional shift ( ADS ) to describe this latter kind of distributional shift , caused by the algorithm ’ s own predictions or behaviour ( Figure 1 ) . ADS are not inherently bad , and are sometimes even desirable . But they can cause problems if they occur unexpectedly . It is typical in machine learning ( ML ) to assume ( e.g . via the i.i.d . assumption ) that ( 2 ) will not happen . However , given the increasing real-world use of ML algorithms , we believe it is important to model and experimentally observe what happens when assumptions like this are violated . This is the motivation of our work . In many cases , including news recommendation , we would consider ( 2 ) a form of cheating—the algorithm changed the task rather than solving it as intended . We care which means the algorithm used to solve the problem ( e.g . ( 1 ) and/or ( 2 ) ) , but we only told it about the ends , so it didn ’ t know not to ’ cheat ’ . This is an example of a specification problem ( Leike et al. , 2017 ; Ortega et al. , 2018 ) : a problem which arises from a discrepancy between the performance metric ( maximize accuracy ) and “ what we really meant ” : in this case , to maximize accuracy via ( 1 ) . Ideally , we ’ d like to quantify the desirability of all possible means , e.g . assign appropriate rewards to all potential strategies and “ side-effects ” , but this is intractable for real-world settings . Using human feedback to learn reward functions which account for such impacts is a promising approach to specifying desired behavior ( Leike et al. , 2018 ; Christiano et al. , 2017 ) . But the same issue can arise whenever human feedback is used in training : one means of improving performance could be to alter human preferences , making them easier to satisfy . Thus in this work , we pursue a complementary approach : managing learners ’ incentives . A learner has an incentive to behave in a certain way when doing so can increase performance ( e.g . accuracy or reward ) . Informally , we say an incentive is hidden when the learner behaves as if it were not present . But we note that changes to the learning algorithm or training regime could cause previously hidden incentives to be revealed , resulting in unexpected and potentially undesirable behaviour . Managing incentives ( e.g . controlling which incentives are hidden/ revealed ) can allow algorithm designers to disincentivize broad classes of strategies ( such as any that rely on manipulating human preferences ) without knowing their exact instantiation.1 The goal of our work is to provide insights and practical tools for understanding and managing incentives , specifically hidden incentives for auto-induced distributional shift : HI-ADS . To study which conditions cause HI-ADS to be revealed , we present unit tests for detecting HI-ADS in supervised learning ( SL ) and reinforcement learning ( RL ) . We also create an environment that models ADS in news recommendation , illustrating possible effects of revealing HI-ADS in this setting . The unit tests both have two means by which the learner can improve performance : one which creates ADS and one which does not . The intended method of improving performance is one that does not induce ADS ; the other is ’ hidden ’ and we want it to remain hidden . A learner `` fails '' the unit test if it nonetheless pursues the incentive to increase performance via ADS . In both the RL and SL unit tests , we find that introducing an outer-loop of meta-learning ( e.g . Population-Based Training ( PBT ) Jaderberg et al . ( 2017 ) ) can lead to high levels of failure . Similarly , recommender systems trained with PBT induce larger drifts in user base and user interests . These results suggest that failure of our unit tests indicates that an algorithm is prone to revealing HI-ADS in other settings . Finally , we propose and test a mitigation strategy we call context swapping . The strategy consists of rotating learners through different environments throughout learning , so that they can ’ t see the results or correlations of their actions in one environment over longer time horizons . This effectively mitigates HI-ADS in our unit test environments , but did not work well in content recommendation experiments . 2 BACKGROUND . 2.1 META-LEARNING AND POPULATION BASED TRAINING . Meta-learning is the use of machine learning techniques to learn machine learning algorithms . This involves instantiating multiple learning scenarios which run in an inner loop ( IL ) , while an outer loop ( OL ) uses the outcomes of the inner loop ( s ) as data-points from which to learn which learning algorithms are most effective ( Metz et al. , 2019 ) . The number of IL steps per OL step is called the interval . Many recent works focus on multi-task meta-learning , where the OL seeks to find learning rules that generalize to unseen tasks by training the IL on a distribution of tasks ( Finn et al. , 2017 ) . Single-task meta-learning includes learning an optimizer for a single task ( Gong et al. , 2018 ) , and adaptive methods for selecting models ( Kalousis , 2000 ) or setting hyperparameters ( Snoek et al. , 2012 ) . For simplicity in this initial study we focus on single-task meta-learning . Population-based training ( PBT ; Jaderberg et al. , 2017 ) is a meta-learning algorithm that trains multiple learners L1 , ... , Ln in parallel , after each interval ( T steps of IL ) applying an evolutionary OL step which consists of : ( 1 ) Evaluate the performance of each learner , ( 2 ) Replace both parameters and hyperparameters of 20 % lowest-performing learners with copies of those from the 20 % high- 1Note removing or hiding an incentive for a behavior is different from prohibiting that behavior , which may still occur incidentally . In particular , not having a ( revealed ) incentive for behaviors that change a human ’ s preferences , is not the same as having a ( revealed ) incentive for behaviors that preserve a human ’ s preferences . The first is often preferable ; we don ’ t want to prevent changes in human preferences that occur “ naturally ” , e.g . as a result of good arguments or evidence . performing learners ( EXPLOIT ) . ( 3 ) Randomly perturb the hyperparameters ( but not the parameters ) of all learners ( EXPLORE ) . 2.2 DISTRIBUTIONAL SHIFT AND CONTENT RECOMMENDATION . In general , distributional shift refers to change of the data distribution over time . In supervised learning with data x and labels y , this can be more specifically described as dataset shift : change in the joint distribution of P ( x , y ) between the training and test sets ( Moreno-Torres et al. , 2012 ; Quionero-Candela et al. , 2009 ) . As identified by Moreno-Torres et al . ( 2012 ) , two common kinds of shift are : ( 1 ) Covariate shift : changing P ( x ) . In the example of content recommendation , this corresponds to changing the user base of the recommendation system . For instance , a media outlet which publishes inflammatory content may appeal to users with extreme views while alienating more moderate users . This self-selection effect ( Kayhan , 2015 ) may appear to a recommendation system as an increase in performance , leading to a feedback effect , as previously noted by Shah et al . ( 2018 ) . This type of feedback effect has been identified as contributing to filter bubbles and radicalization ( Pariser , 2011 ; Kayhan , 2015 ) . ( 2 ) Concept shift : changing P ( y|x ) . In the example of content recommendation , this corresponds to changing a given user ’ s interest in different kinds of content . For example , exposure to a fake news story has been shown to increase the perceived accuracy of ( and thus presumably future interest in ) the content , an example of the illusory truth effect ( Pennycook et al. , 2019 ) . For further details on such effects in content recommendation , see Appendix 8 . 3 AUTO-INDUCED DISTRIBUTION SHIFT ( ADS ) . Auto-induced distribution shift ( ADS ) is distributional shift caused by an algorithm ’ s behaviour . This is in contrast to distributional shift which would happen even if the learner were not present - e.g . for a crash prediction algorithm trained on data from the summer , encountering snowy roads is an example of distributional shift , but not auto-induced distributional shift ( ADS ) . a1 a2 s1 s2 r1 r2 ( a ) RL : Incentives for ADS are present ; pursuing them is desirable a1 a2 s1 s2 r1 r2 ( b ) Myopic RL : Incentives for ADS are present ; pursuing them is undesirable ŷ1 ŷ2 x1 x2 ` 1 ` 2 ( c ) SL with ADS : Incentives for ADS are present ; pursuing them is undesirable ŷ1 ŷ2 x1 x2 ` 1 ` 2 ( d ) SL with i.i.d . data : Incentives for ADS are absent Figure 2 : The widely studied problems of reinforcement learning ( RL ) with state s , action a , reward r tuples , and i.i.d . supervised learning ( SL ) with inputs x , predictions ŷ and loss l ( a , d ) are free from incentive problems . We focus on cases where there are incentives present which the learner is not meant to pursue ( b , c ) . Lines show paths of influence . The learner may have incentives to influence any nodes descending from its action , A , or prediction , ŷ . Which incentives are undesirable ( orange ) or desirable ( cyan ) for the learner to pursue is context-dependent . We emphasize that ADS are not inherently bad or good ; often ADS can even be desirable : consider an algorithm meant to alert drivers of imminent collisions . If it works well , such a system will help drivers avoid crashing , thus making self-refuting predictions which result in ADS . What separates desirable and undesirable ADS ? The collision-alert system alters its data distribution in a way that is aligned with the goal of fewer collisions , whereas the news manipulation results in changes that are misaligned with the goal of better predicting existing users ’ interests ( Leike et al. , 2018 ) . In reinforcement learning ( RL ) , ADS are typically encouraged as a means to increase performance . On the other hand , in supervised learning ( SL ) , the i.i.d . assumption precludes ADS in theory . In practice , however , the possibility of using ADS to increase performance ( and thus an incentive to do so ) often remains . For instance , this occurs in online learning . In our experiments , we explicitly model such situations where i.i.d . assumptions are violated : We study the behavior of SL and myopic RL algorithms , in environments designed to include incentives for ADS , in order to understand when incentives are effectively hidden . Fig . 2 contrasts these settings with typical RL and SL .

Summary: This paper introduces the concept of auto-induced distributional shift (ADS), and argues that some meta learning and reinforcement learning algorithms have the incentives to change the distribution so that the problem is easier to solve. The paper presents unit tests to detect hidden incentives for auto-induced distributional shift (HI-ADS) and also proposes context swapping to reduce the distributional shift. Experiments on Population-Based Training (PBT) show that PBT reveals HI-ADS in unit tests and context swapping mitigates the distributional shift for PBT.