Hidden incentives for self-induced distributional shift

1 INTRODUCTION . Consider a household robot , one of whose duties is to predict when its owner will ask it for coffee . We would like the robot to notice its owners preference for having coffee in the morning , but we would not want the robot to prevent its owner from sleeping late just because the robot is unsure if the owner will still want coffee if they wake up in the afternoon . While doing so would result in a better prediction , such a strategy is cheating - by changing the task rather than solving the task as intended . More specifically , waking the owner is an example of what we call self-induced distributional shift ( SIDS ) , as it changes the distribution of inputs to the robot ’ s coffee prediction algorithm . SIDS is not necessarily undesirable : consider an algorithm meant to alert drivers of imminent collisions . If it works well , such a system will help drivers avoid crashing , thus making self-refuting predictions which result in SIDS . What separates this example from the coffee robot that disturbs its owner ’ s sleep ? The collision-alert system alters its data distribution in a way that is aligned with the goal of fewer collisions , whereas the coffee robot ’ s strategy results in changes that are misaligned with the goal of good coffee-timing ( Leike et al. , 2018 ) . This makes it an example of a specification problem ( Leike et al. , 2017 ; Ortega & Maini , 2018 ) : we did not intend the robot to ensure its predictions were good using such a strategy , yet a naive specification ( e.g . maximizing likelihood ) incentivized that strategy . Ideally , we ’ d like to specify which kinds of SIDS are acceptable , i.e . the means by which a learner is intended or allowed to influence the world in order to achieve its ’ ends ( i.e . increase its performance ) , but doing so in full generality can be difficult . An alternative , more tractable problem which we address in this work is to accept the possibility of SIDS , but to carefully manage incentives for SIDS . Informally , a learner has an incentive to behave in a certain way when doing so can increase its performance ( e.g . higher accuracy , or increased reward ) . When meta-learning optimizes over a longer time horizon , or using a different algorithm , than the original “ inner loop ” learner , this can reveal new incentives for SIDS that were not apparent in the original learner ’ s behavior . We call these hidden incentives for distributional shift ( HIDS ) , and note that keeping HIDS hidden can be important for achieving aligned behavior . Notably , even in the absence of an explicit meta-learning algorithm machine learning practitioners employ “ manual meta-learning ” , also called “ grad student descent ” ( Gencoglu et al. , 2019 ) in the iterative process of algorithm design , model selection , hyperparameter tuning , etc . Considered in this broader sense , meta-learning seems indispensable , making HIDS relevant for all machine learning practitioners . A real-world setting where incentives for SIDS could be problematic is content recommendation : algorithmically selecting which media or products to display to the users of a service . For example ( see Figure 1 ) , a profit-driven algorithm might engage in upselling : persuading users to purchase or click on items they originally had no interest in . Recent media reports have described ‘ engagement ’ - ( click or view-time ) driven recommendation services such as YouTube contributing to viewer radicalization ( Roose , 2019 ; Friedersorf , 2018 ) . A recent study supports these claims , finding that many YouTube users “ systematically migrate from commenting exclusively on milder content to commenting on more extreme content ” ( Ribeiro et al. , 2019 ) .1 See Appendix 1 for a review of real-world issues related to content recommendation . Our goal in this work is to show both ( 1 ) that meta-learning can reveal HIDS , and ( 2 ) that this means applying meta-learning to a learning scenario not only changes the way in which solutions are searched for , but also which solutions are ultimately found . Our contributions are as follows : 1 . We identify and define the phenomena of SIDS ( self-induced distributional shift ) and HIDS ( hidden incentives for distributional shift ) . 2 . We create two simple environments for studying identifying and studying HIDS : a “ unit test ” based on the Prisoner ’ s Dilemma , and a content recommendation environment which disentangles two types of SIDS . 3 . We demonstrate experimentally that meta-learning reveals HIDS in these environments , yielding agents that achieve higher performance via SIDS , but may follow sub-optimal policies . 4 . We propose and test a mitigation strategy based on swapping learners between environments in order to reduce incentives for SIDS . 2 BACKGROUND . 2.1 DISTRIBUTIONAL SHIFT AND CONTENT RECOMMENDATION . In general , distributional shift refers to change of the data distribution over time . In supervised learning with data x and labels y , this can be more specifically described as dataset shift : change in the joint distribution of P ( x , y ) between the training and test sets ( Moreno-Torres et al. , 2012 ; Quionero-Candela et al. , 2009 ) . As identified by Moreno-Torres et al . ( 2012 ) , two common kinds of distributional shift are : 1 . Covariate shift : changing P ( x ) . In the context of content recommendation , this corresponds to changing the user base of the recommendation system . For instance , a media outlet which publishes inflammatory content may appeal to users with extreme views while alienating more moderate users . This self-selection effect ( Kayhan , 2015 ) may appear to a recommendation system as an increase in performance , leading to a feedback effect , as previously noted by Shah et al . ( 2018 ) . This type of feedback effect has been identified as contributing to filter bubbles and radicalization ( Pariser , 2011 ; Kayhan , 2015 ) . We observe this type of change in our experiments , as shown in Figure 1 . 1 The authors argue that commenting on a video is a good proxy for supporting its viewpoint , since only 5 out of 900 comments they checked opposed the viewpoint of the video commmented on . 2 . Concept shift : changing P ( y|x ) . In the context of content recommendation , this corresponds to changing a given user ’ s interest in different kinds of content . For example , exposure to a fake news story has been shown to increase the perceived accuracy of ( and thus presumably the interest in ) the story , an example of the illusory truth effect ( Pennycook et al. , 2019 ) . 2.2 META-LEARNING AND POPULATION BASED TRAINING . Meta-learning is the use of machine learning techniques to learn machine learning algorithms . This generally involves instantiating multiple learning scenarios which run in an inner loop ( IL ) , while an outer loop ( OL ) uses the outcomes of the inner loop ( s ) as data-points from which to learn which learning algorithms are most effective ( Metz et al. , 2019 ) . The number of IL steps per OL step is called the interval of the OL . Many recent works have focused on multi-task meta-learning where the OL seeks to find learning rules that generalize to unseen tasks by training the IL on a distribution of tasks - this is often used as an approach to one- or few-shot learning , e.g . Finn et al . ( 2017 ) ; Ren et al . ( 2018 ) , or transfer learning , e.g . Andrychowicz et al . ( 2016 ) . Single-task meta-learning includes learning an optimizer for a single task , e.g . Gong et al . ( 2018 ) , adaptive methods for selecting models , e.g . Kalousis ( 2000 ) , or for setting hyperparameters , e.g . Snoek et al . ( 2012 ) . For simplicity in this initial study we focus on single-task meta-learning . Population-based training ( PBT ) ( Jaderberg et al. , 2017 ) is a meta-learning algorithm that trains multiple learners L1 , ... , Ln in parallel , after each interval ( T steps of IL ) applying an evolutionary OL step which consists of : 1 . Evaluate the performance of each learner , 2 . Replace both parameters and hyperparameters of low-performing ( bottom 20 % ) learners with copies of those from randomly chosen high-performing ( top 20 % ) learners ( EXPLOIT ) , 3 . Randomly perturb the hyperparameters ( but not the parameters ) of all learners ( EXPLORE ) . Two distinctive features of PBT ( compared with other hyperoptimization methods , such as Bayesian optimization ( Snoek et al. , 2012 ) ) are notable for us because they give the OL more control over the learning process : 1 . PBT applies OL optimization to parameters , not just hyperparameters . This means the OL can directly select for parameters which lead to SIDS , instead of only being able to influence parameter values via hyperparameters , which may be much more limiting . 2 . PBT uses multiple OL steps within a single training run . This gives the OL more overall influence over the dynamics and outcome of the training run . 2.3 SPECIFICATION AND INCENTIVES . We define specification as the process of a ( typically human ) designer instantiating a learning algorithm in a real-world learning scenario ( see Appendix 2 for formal definitions ) . A specification problem occurs when the outcome of a learning scenario differs from the intentions of the designer . Specification is often viewed as concerned solely with the choice of performance metric , and indeed researchers often select learners solely on the basis of performance . However , our work emphasizes that the choice of learning algorithm is also an aspect of specification , as noted by Ortega & Maini ( 2018 ) . In particular , we consider this choice from the point of view incentives , similarly to Everitt et al . ( 2019 ) . Their work focused on identifying which incentives exist , but we note that incentives may exist and yet not be pursued by a learner ; for example , in supervised learning , there is an incentive to overfit the test set in order to increase test performance , but algorithms are designed to not do that . We thus distinguish between the existence of an incentive in a learner ’ s operational context and its presence in a learner ’ s objective , or revealed specification ( Ortega & Maini , 2018 ) , which is what a learner is “ trying ” to accomplish . Given an incentive that is present in the operational context , we say it is hidden from a learner if it does not appear in the objective , and revealed if it does . 3 SELF-INDUCED DISTRIBUTION SHIFT ( SIDS ) AND HIDDEN INCENTIVES FOR DISTRIBUTIONAL SHIFT ( HIDS ) . 3.1 SIDS . To formally define SIDS , we assume there exists some reference data distribution , which is the distribution of data that the learner would encounter “ by default ” . This is a standard assumption for classification problems ( Moreno-Torres et al. , 2012 ) ; in reinforcement learning , the reference distribution could be the initial distribution over states , or the distribution over states which results from following some reference policy . We say that SIDS occurs whenever the behavior ( e.g . actions or predictions , or mere existence ) , of the learner leads it to encounter a distribution other than this reference distribution . This definition excludes distributional shift which would happen even if the learner were not present - e.g . for a crash prediction algorithm trained on data from the summer , snowy roads in the winter are an example of distributional shift , but not self-induced distributional shift ( SIDS ) . In order to highlight the phenomenon of SIDS , we distinguish between the ( often implicit ) assumptions of the machine learning algorithm ( e.g . the i.i.d . assumption ) , vs. the model of the environments in which the algorithm is trained/deployed ( e.g . our synthetic content recommendation environment ) . This is formalized in Appendix 2 . This distinction allows us to explicitly model situations in which the assumptions of a learning algorithm are violated . For instance , in Sec . 4.2 we explicitly model a partially observable environment whose underlying state determines the data distribution of the examples that a standard supervised learning algorithm observes at each time-step .

Summary: The authors study the phenomena of self-introduced distributional shift. They define the term along with the term hidden incentives for distributional shift. The latter describes factors that motivate the learner to change the distribution in order to achieve a higher performance. The authors study both phenomena in two domains (one being a prisoner dilemma and the other a recommender system) and show how meta-learning reveals the hidden incentives for distributional shift. They then propose an approach based on swapping learners between environments to reduce self introduced distributional shift.