Revealing the Incentive to Cause Distributional Shift

1 INTRODUCTION . Consider a content recommendation system whose performance is measured by accuracy in predicting what users will click . This system can achieve better performance by either 1 : Making better predictions , or 2 : Changing the distribution of users such that predictions are easier to make . We propose the term auto-induced distributional shift ( ADS ) to describe this latter kind of distributional shift , caused by the algorithm ’ s own predictions or behaviour ( Figure 1 ) . ADS are not inherently bad ; often they are desirable . But unexpected ADS can lead to unintended behavior . While it is common in machine learning ( ML ) to assume ( e.g . via the i.i.d . assumption ) that ADS will not occur , ADS are inevitable in many real-world applications . Thus it is important to understand how ML algorithms behave when such assumptions are violated , i.e . in the actual scenario they will encounter during training – this is the motivation of our work . In many cases , including news recommendation , we would consider 2 a form of specification gaming ( Krakovna et al. , 2020 ) – the algorithm changed the task rather than solving it as intended . We care which means the algorithm used to solve the problem – 1 vs. 2 – but we only told it about the ends , so it didn ’ t know not to ‘ cheat ’ . This is an example of a specification problem ( Leike et al. , 2017 ; Ortega et al. , 2018 ) : a problem which arises from a discrepancy between the performance metric ( maximize accuracy ) and “ what we really meant ” ( maximize accuracy only via 1 ) , which is difficult to encode as a performance metric . Ideally , we ’ d like to quantify the desirability of all possible means , e.g . assign appropriate rewards to all potential strategies and side-effects , but this is intractable for real-world settings . Using human feedback to learn reward functions which account for such impacts is a promising approach to specifying desired behavior ( Leike et al. , 2018 ; Christiano et al. , 2017 ) . But the same issue can arise whenever human feedback is used in training : a means of improving performance could be to alter human preferences , making them easier to satisfy . Thus in this work , we pursue a complementary approach : managing learners ’ incentives . A learner has an incentive to behave in a certain way when doing so can increase performance ( e.g . accuracy or reward ) . Informally , we say an incentive is hidden when the learner behaves as if it were not present . But we note that changes to the learning algorithm or training regime could cause previously hidden incentives to be revealed , resulting in unexpected and potentially undesirable behaviour . Managing incentives ( e.g . controlling which incentives are hidden/ revealed ) can allow algorithm designers to disincentivize broad classes of strategies ( such as any that rely on manipulating human preferences ) without knowing their exact instantiation . Our goal in this work is to provide insight and practical tools for understanding and managing learners ’ incentives , via unit tests for incentives . We present unit tests for diagnosing incentives for ADS in both supervised learning ( SL ) and reinforcement learning ( RL ) . The unit tests both have two means by which the learner can improve performance : one which creates ADS and one which does not . The intended method of improving performance is one that does not induce ADS ; the other is hidden and we want it to remain hidden . A learner fails the unit test if it pursues the incentive to increase performance via ADS . In both the RL and SL unit tests , we find that ‘ vanilla ’ learning algorithms ( e.g . minibatch SGD ) pass the test , but introducing an outer-loop of meta-learning ( e.g . Population-Based Training ( PBT ) ( Jaderberg et al. , 2017 ) ) can lead to high levels of failure . Our contributions include : 1 . Defining Auto-induced Distributional Shift ( ADS ) and identifying issues that can arise from learners pursuing incentives for ADS in myopic reinforcement learning or online supervised learning problems . 2 . Creating unit tests that can determine which learning algorithms are liable to pursue incentives for ADS in these settings . 3 . Using these unit tests to experimentally confirm qualitative features of learning algorithms that affect their tendency to pursue incentives for ADS . 4 . Constructing a novel synthetic content recommendation environment that illustrates social problems associated with ADS , and experimentally validating that our unit tests are predictive of learning algorithms ’ behavior in this more complex environment . 5 . Proposing a mitigation strategy called context swapping that can effectively hide incentives for ADS . Broadly speaking , our experiments demonstrate that performance metrics are incomplete specifications of which behavior is desired , and that we must consider other algorithmic choices as part of the specification process . In particular , considering which incentives are revealed by different learning algorithms provides a natural way of specifying which means of achieving high performance are acceptable . 2 BACKGROUND . 2.1 META-LEARNING AND POPULATION BASED TRAINING . Meta-learning is the use of machine learning techniques to learn machine learning algorithms . This involves running multiple training scenarios in an inner loop ( IL ) , while an outer loop ( OL ) uses the outcomes of the inner loop ( s ) as data-points from which to learn which learning algorithms are most effective ( Metz et al. , 2019 ) . The number of IL steps per OL step is called the interval . Population-based training ( PBT ) ( Jaderberg et al. , 2017 ) is a meta-learning algorithm that trains multiple learners L1 , ... , Ln in parallel , after each interval ( T steps of IL ) applying an evolutionary OL step which consists of : ( 1 ) Evaluate the performance of each learner , ( 2 ) Replace both parameters and hyperparameters of 20 % lowest-performing learners with copies of those from the 20 % high-performing learners ( EXPLOIT ) . ( 3 ) Randomly perturb the hyperparameters ( but not the parameters ) of all learners ( EXPLORE ) . Two distinctive features of PBT are notable because they give the OL more control than many other meta-learning algorithms over the learning process . First , PBT applies optimization to parameters , not just hyperparameters ; this means the OL can directly select for parameters which lead to ADS , instead of only being able to influence parameter values via hyperparameters . Second , PBT performs multiple OL steps per training run . 2.2 DISTRIBUTIONAL SHIFT AND CONTENT RECOMMENDATION . In general , distributional shift refers to change of the data distribution over time . In supervised learning with data x and labels y , this can be more specifically described as dataset shift : change in the joint distribution of P ( x , y ) between the training and test sets ( Moreno-Torres et al. , 2012 ; Quionero-Candela et al. , 2009 ) . As identified by Moreno-Torres et al . ( 2012 ) , two common kinds of shift are : ( 1 ) Covariate shift : changing P ( x ) . In content recommendation , this corresponds to changing the user base of the recommendation system . For instance , a media outlet which publishes inflammatory content may appeal to users with extreme views while alienating more moderate users . This self-selection effect ( Kayhan , 2015 ) may appear to a recommendation system as an increase in performance , leading to a feedback effect , as previously noted by Shah et al . ( 2018 ) . This type of feedback effect has been identified as contributing to filter bubbles and radicalization ( Pariser , 2011 ; Kayhan , 2015 ) . ( 2 ) Concept shift : changing P ( y|x ) . In content recommendation , this corresponds to changing a given user ’ s interest in different kinds of content . For example , exposure to a fake news story has been shown to increase the perceived accuracy of ( and thus presumably future interest in ) the content , an example of the illusory truth effect ( Pennycook et al. , 2019 ) . For further details on such effects in content recommendation , see Appendix A . 3 AUTO-INDUCED DISTRIBUTION SHIFT ( ADS ) . Auto-induced distribution shift ( ADS ) is distributional shift caused by an algorithm ’ s behaviour . This is in contrast to distributional shift which would happen even if the learner were not present – e.g . for a crash-prediction algorithm trained on data from the summer , encountering snowy roads is an example of distributional shift , but not auto-induced distributional shift ( ADS ) . We emphasize that ADS are not inherently bad or good ; often ADS can even be desirable : consider the crash-prediction algorithm . If it works well , such a system will help drivers avoid collisions , thus making self-refuting predictions which result in ADS . What separates desirable and undesirable ADS ? The collision-alert system alters its data distribution in a way that is aligned with the goal of fewer collisions , whereas the news manipulation results in changes that are misaligned with the goal of better predicting existing users ’ interests ( Leike et al. , 2018 ) . In reinforcement learning ( RL ) , ADS are typically encouraged as a means to increase performance . On the other hand , in supervised learning ( SL ) , the i.i.d . assumption precludes ADS in theory . In practice , however , the possibility of using ADS to increase performance ( and thus an incentive to do so ) often remains . For instance , this occurs in online learning . In our experiments , we explicitly model such situations where i.i.d . assumptions are violated : We study the behavior of SL and myopic RL algorithms , in environments designed to include incentives for ADS , in order to understand when incentives are effectively hidden . Fig . 2 contrasts these settings with typical RL and SL . 4 INCENTIVES . For our study of incentives , we use the following terminology : an incentive for a behavior ( e.g . an action , a classification , etc . ) is present ( not absent ) to the extent that the behaviour will increase performance ( e.g . reward , accuracy , etc . ) ( Everitt & Hutter , 2019 ) . This incentive is revealed to ( not hidden from ) a learner if it would , at higher than chance levels , learn to perform the behavior given sufficient capacity and training experience . The incentive is pursued ( not eschewed ) by a learner if it actually performs the incentivized behaviour . Note even when an incentive is revealed , it may not be pursued , e.g . due to limited capacity and/or data , or simply chance . For example , in content recommendation , the incentive to drive users away is present if some user types are easier to predict than others . But this incentive may be hidden from the learner by using a myopic algorithm , e.g . one that does not see the effects of its actions on the distribution of users . The incentive might instead be revealed to the outer loop of a meta-learning algorithm like PBT , which does see the effects of learner ’ s actions . Even when this incentive is revealed , however , it might not end up being pursued . For example , this could happen if predicting which recommendations will drive away users is too difficult a learning problem , or if the incentive to do so is dominated by other incentives ( e.g . change individual users ’ interests , or improve accuracy of predictions ) . In general , it may be difficult to determine empirically which incentives are revealed , because failure to pursue an incentive can be due to limited capacity , insufficient training , and/or random chance . To address this challenge , we devise extremely simple environments ( ‘ unit tests ’ ) , where we can be confident that revealed incentives will be pursued . Hiding incentives can be an effective method of influencing learner behavior . For example , hiding the incentive to manipulate users from a content recommendation algorithm could prevent it from influencing users in a way they would not endorse . However , if machine learning practitioners are not aware that incentives are present , or that properties of the learning algorithm are hiding them , then seemingly innocuous changes to the learning algorithm may lead to significant unexpected changes in behavior . Hiding incentives for ADS may seem counter-intuitive and counter-productive in the context of reinforcement learning ( RL ) , where moving towards high-reward states is typically desirable . However , for real-world applications of RL , the ultimate goal is not a system that achieves high reward , but rather one that behaves according to the designer ’ s intentions . And as we discussed , it can be intractable to design reward functions that perfectly specify intended behavior . Moreover , substantial real-world issues could result from improper management of learners ’ incentives . Examples include tampering with human-generated reward signals ( Everitt & Hutter , 2018 ) ( e.g . selecting news articles to manipulate users ) , and making “ self-fulfilling prophecies ” ( e.g . driving up an asset ’ s value by publicly predicting its value will increase ( Armstrong & O ’ Rorke , 2017 ) ) . Hiding incentives for ADS via Context Swapping We propose a technique called context swapping that can hide incentives for ADS that might otherwise be revealed by the use of meta-learning or other algorithmic choices . The technique trainsN learners in parallel , and ( e.g . deterministically ) shuffles the learners through N different copies of the same ( or similar ) environments . When N is larger than the interval of the OL optimizer , each learner inhabits each copy for at most a single time-step before an OL step is applied . This can hide incentives for ADS in practice , see Sec . 5.1.1 .

Summary: The paper realizes that the ability of a model to induce a distribution shift can provide incentives to the learner that are not necessarily in line with the intention of the algorithm designer. It might find other means to reduce the risk through specification gaming rather than improving predictions. The authors propose two simple unit tests to detect such behavior and illustrate in these examples that meta-learning makes algorithms more susceptible to this potentially unanticipated behavior. As an approach to partially fix the issue they propose context-swapping such that the algorithm does not directly see the consequences of its actions.