Hidden incentives for self-induced distributional shift
Authors: David Scott Krueger
Tegan Maharaj
Shane Legg
Jan Leike
Venue:  (2020)
Keywords: distributional shift
safety
incentives
specification
content recommendation
reinforcement learning
online learning
ethics

Abstract: Decisions made by machine learning systems have increasing influence on the world. Yet it is common for machine learning algorithms to assume that no such influence exists. An example is the use of the i.i.d. assumption in online learning for applications such as content recommendation, where the (choice of) content displayed can change users' perceptions and preferences, or even drive them away, causing a shift in the distribution of users. Generally speaking, it is possible for an algorithm to change the distribution of its own inputs. We introduce the term self-induced distributional shift (SIDS) to describe this phenomenon. A large body of work in reinforcement learning and causal machine learning aims to deal with distributional shift caused by deploying learning systems previously trained offline. Our goal is similar, but distinct: we point out that changes to the learning algorithm, such as the introduction of meta-learning, can reveal hidden incentives for distributional shift (HIDS), and aim to diagnose and prevent problems associated with hidden incentives. We design a simple  environment as a "unit test" for HIDS, as well as a content recommendation environment which allows us to disentangle different types of SIDS.  We demonstrate the potential for HIDS to cause unexpected or undesirable behavior in these environments, and propose and test a mitigation strategy. 

Under review as a conference paper at ICLR 2020
HIDDEN INCENTIVES FOR SELF-INDUCED DISTRIBU- TIONAL SHIFT
Anonymous authors Paper under double-blind review
ABSTRACT
Decisions made by machine learning systems have increasing influence on the world. Yet it is common for machine learning algorithms assume that such influence exists. An example is the use of the i.i.d. assumption in online learning for applications such content recommendation, where the (choice of) content displayed can change users' perceptions and preferences, or even drive them away, causing shift in the distribution of users. Generally speaking, an algorithm to change the distribution of its own inputs. We introduce the term self-induced distributional shift (SIDS) to describe this phenomenon. A large body of work in reinforcement learning and causal machine learning aims to deal with distributional shift caused by deploying learning systems previously trained offline. Our goal is similar, but distinct: we point out that changes to the learning algorithm, such as the introduction of meta-learning, can reveal hidden incentives for distributional shift (HIDS). and aim to diagnose and prevent problems as- sociated with hidden incentives. We design a simple environment as a 'unit test' for HIDS, well content recommendation environment which allows us disentangle different types of SIDS. We demonstrate the potential for HIDS to cause unexpected or undesirable behavior in these environments, and propose and test a mitigation strategy.
possible for
INTRODUCTION
Consider a household robot, one of whose duties is to predict when its owner will ask it for coffee. We would like the robot to notice its owners preference for having coffee in the morning, but we would not want the robot to prevent its owner from sleeping late just because the robot is unsure if the owner will still want coffee if they wake up in the afternoon. While doing so would result in better prediction, such a strategy is cheating by changing the task rather than solving the task as intended. More specifically, waking the owner is an example of what we call self-induced distributional shift (SIDS), as it changes the distribution of inputs to the robot's coffee prediction algorithm.
SIDS is not necessarily undesirable: consider an algorithm meant to alert drivers of imminent collisions. If it works well, such a system will help drivers avoid crashing. thus making self-refuting predictions which result in SIDS. What separates this example from the coffee robot that disturbs its owner's i sleep? The collision-alert system alters its data distribution in way that is aligned with the goal of fewer collisions, whereas the coffee robot's strategy results in changes that are misaligned with the goal of good coffee-timing (Leike al. 2018).
This makes it an example of specification problem (Leike et al., 2017 Ortega Maini 2018): we did not intend the robot to ensure its predictions were good using such strategy, yet a naive specification (e.g. maximizing likelihood) incentivized that strategy. Ideally, we'd like to specify which kinds of SIDS are acceptable, i.c. the means by which learner is intended or allowed to influence the world in order achieve its' ends (i.e. increase its performance), but doing so in full generality can be difficult. An alternative, more tractable problem which we address in this work is to accept the possibility of SIDS, but to carefully manage incentives for SIDS.
Informally, a learner has an incentive to behave in certain way when doing so can increase its performance (e.g. higher accuracy, or increased reward). When meta-learning optimizes over a longer time horizon, or using a different algorithm, than the original "inner loop" learner, this can reveal new incentives for SIDS that were not apparent in the original learner's behavior. We call these hidden incentives for distributional shift (HIDS), and note that keeping HIDS hidden can be important for achieving aligned behavior. Notably, even in the absence of an explicit meta-learning algorithum machine learning practitioners employ "manual meta- leta-learning". also called "grad student descent" (Gencoglu et al. 2019) in the iterative process of algorithm design, model selection, hyperparameter

---

Under review as a conference paper at ICLR 2020
tuning, etc. Considered in this broader sense, meta-learning seems indispensable, making HIDS relevant for all machine learning practitioners.
A real-world setting where incentives for SIDS could be problematic is content recommendation: algorithmically selecting which media or products to display to the users a service. For example (see Figure D. profit-driven algorithm might engage in upselling: persuading users purchase or click on items they originally had no interest in. Recent media reports have described 'engagement' (click or view-time) driven recommendation services such as YouTube contributing to viewer radicalization (Roose 2019; Friedersorf] 2018). A recent study supports these claims, finding that many You' Tube users "systematically migrate from commenting exclusively on milder content to commenting on more extreme content" (Ribeiro et al., 2019) See Appendix for a review of real-world issues related to content recommendation.
Our goal in this work is to show both (1) that meta-learning can reveal HIDS, and (2) that this means applying meta-learning to learning scenario not only changes the way in which solutions are searched for, but also which solutions are ultimately found. Our contributions are as follows:
1. We identify and define the phenomena of SIDS (self-induced distributional shift) and HIDS (hidden incentives for distributional shift).
2. We create simple environments for studying identifying and studying HIDS: "unit test" based on the Prisoner's Dilemma, and a content recommendation environment which disentangles two types of SIDS.
3. We demonstrate experimentally that meta-learning reveals HIDS in these enviromments, yielding agents that achieve higher performance via SIDS, but may follow sub-optimal policies.
4. We propose and test mitigation strategy based on swapping learners between environments in order to reduce incentives for SIDS.
Desembtt on of cver time tno SIS1
Disetiitonn
time
-000
1900
Figure 1: Distributions of users over time. Left: A distribution which remains constant over time, following the i.i.d assumption. Right: Self-induced distributional shift (SIDS) results in a change in the distribution of users in our content recommendation environment (see Section 4.2 for details).
2 BACKGROUND
2.1 DISTRIBUTIONAL SHIFT AND CONTENT RECOMMENDATION
In general, distributional shift refers to change of the data distribution over time. In supervised learning with data x and labels y. this can be more specifically described as dataset shift: change in the joint distribution of P(x, y) between the training and test sets (Moreno-Toreee et al. 2012 Caanerrr-Candlla et al. 2009). As identified by Mooeno-Tooree et al. (202)), two common kinds of distributional shift are:
1. Covariate shift: changing P(x). In the context of content recommendation, this corresponds to changing the user base of the recommendation system. For instance, a media outlet which publishes inflammatory content may appeal to users with extreme views while alienating more moderate users. This self-selection effect (Kayhan, 2015) may appear to a recommendation system an increase in performance, leading
feedback effect, previously noted by Shah et al. (2018). This type of feedback effect has been identified as contributing to filter bubbles and radicalization (Pariser 2011 Kayhan 2015). We observe this type of change in our experiments, as shown in Figure]
1 The authors argue that commenting on a video is good proxy for supporting its viewpoint, since only 5 of 900 comments they checked opposed the viewpoint of the video commmented
2

---

1. Evaluate the performance of each learner, Under review as a conference paper at ICLR 2020
2. Concept shift: changing P(y(x). In the context of content recommendation, this corresponds to changing given user's interest in different kinds of content. For example, exposure to a fake news story has been shown to increase the perceived accuracy of (and thus presumably the interest in) the story, an example of the illusory truth effect (Pennycook et al. 2019).
2.2 META-LEARNING AND POPULATION BASED TRAINING
Meta-learning is the use of machine learning techniques to learn machine learning algorithms. This generally involves instantiating multiple learning scenarios which run in inner loop (II.), while an outer loop (OL) uses the outcomes of the inner loop(s) as data-points from which to learn which learning algorithms are most effective (Metz et al. 2019). The number of IL steps per OL step is called the interval of the OL.
Many recent works have focused on multi-task meta-learning where the OL seeks to find learning rules that generalize to unseen tasks by training the IL. on distribution of tasks this is often used as an approach to one- or few-shot learning, e.g. FFin et al. (2017): Ren et al. (2018), or transfer learning, e.g. Andrychowicz et al. (2016). Single-task meta-learning includes learning an optimizer for single task, e.g. Gong et al. (2018), adaptive methods for selecting models, e.g. Kalousis (2000), or for setting hyperparameters, e.g. Snock et al. (2012). For simplicity in this initial study we focus on single-task meta-leanaan meta-learning
Population- based training (PBT) (Jaderberg et al. 2017 is meta-learning algorithm that trains multiple learners I,..., Ie in parallel, after each interval (T steps of IL. applying an evolutionary OL step which consists of:
Replace both parameters and hyperparameters low-performing (bottom 20%) learners with
copies of those from randomly chosen high-perforning (top 20%) learners (EXPLOIT)
3. Randomly perturb the hyperparameters (but not the parameters) of all learners (EXPLORE)) Two distinctive features of PBT (compared with other hyperoptimization methods, such Bayesian optimization (Snock et al. 2012) are notable for us because they give the OL more control over the learning process:
1. PBT applics OL optimization to parameters, not just hyperparameters. This means the OL can directly select for parameters which lead to SIDS, instead of only being able influence parameter values via hyperparameters, which may be much more limiting. PBT uses multiple OL steps within a single training run. This gives the OL more overall influence over the dynamics and outcome of the training run.
2.3 SPECIFICATION AND INCENTIVES
We define specification the process of a (typically human) designer instantiating a learning algorithm in real-world learning scenario (see Appendix S for formal definitions). specification problem occurs when the outcome of learning scenario differs from the intentions of the designer. Specification is often viewed concerned solely with the choice of performance metric, and indeed researchers often select learners solely on the basis of performance. However, our work emphasizes that the choice of learning algorithm is also an aspect of specification, as noted by Ortega Maini 2018))
In particular, we consider this choice from the point of view incentives, similarly to Everitt et al. (2019). Their work focused on identifying which incentives exist, but note that incentives may exist and yet not be pursued by learner; for example, in supervised learning, there is an incentive to overfit the test set in order increase performance, but algorithms We thus distinguish between the existence of an incentive a learner's operational context and its presence in learner's objective, revealed specification (Ortega & Maini (218)), which is what learner is "trying" to accomplish. Given an incentive that is present in the operational context, we say it is hidden from a learner if it does not appear in the objective, and revealed if does.
designed to not do that.
SELF-INDUCED DISTRIBUTION SHIFT (SIDS) AND HIDDEN INCENTIVES FOR DISTRIBUTIONAL SHIFT (HIDS)
3.1 SIDS
To formally define SIDS, we assume there exists some reference data distribution, which is the distribution of data that the learner would encounter "by default".. This is standard assumption
3

---

Under review as a conference paper at ICLR 2020
for classification problems (Moreno- Torres al. 2012); in reinforcement learning, the reference distribution could be the initial distribution over states, or the distribution over states which results from following some reference policy. We say that SIDS occurs whenever the behavior (e.g. actions or predictions, or mere existence), of the learner leads it to encounter a distribution other than this reference distribution. This definition excludes distributional shift which would happen even if the learner were not present c.g. for a crash prediction algorithm trained on data from the summer, snowy roads in the winter are an example of distributional shift, but not self-induced distributional shift (SIDS).
In order to highlight the phenomenon of SIDS, we distinguish between the (often implicit) assump- tions of the machine learning algorithm (e.g. the i..d. assumption), vs. the model of the environments in which the algorithm is trained/deployed (e.g. our synthetic content recommendation environment). This is formalized in Appendix 2 This distinction allows us to explicitly model situations in which the assumptions of learning algorithm are violated. For instance, in Sec. 4.2 we explicitly model partially observable environment whose underlying state determines the data distribution of the examples that a standard supervised learning algorithm observes at each time-step.
3.2 HIDS
Referring to Section 2.3 we say that incentives for SIDS are hidden if they are not part of the objective of learner. Like SIDS, HIDS are not necessarily good or bad. Rather, our point that designers should be cognizant of which incentives exist, and whether they are hidden or revealed to learner. More specifically, changing the learning algorithm can reveal incentives that were previously hidden, leading learners to adopt unanticipated and potentially undesirable strategies for maximizing performance. For instance, by optimizing for performance after a sequence of inner loop updates, meta-learning can fail distinguish between solving the task intended and making the task easier via SIDS, and thus can reveal hidden incentives for distributional shift (HIDS).
In many settings, such as reinforcement learning (uutonn& & Barto 1998), , learners are intended to increase performance via SIDS. For prediction tasks, on the other hand, learners are typically not meant to seek distributional shift, even if there an incentive to do so, as we illustrate with the coffee robot example in the introduction. And even in reinforcement learning, SIDS can be undesirable, as we illustrate in Sec.
3.3 CONTEXT SWAPPING: A MITIGATION TECHNIQUE
We propose a technique called context swapping for mitigating HIDS revealed by meta-learning. The idea of context swapping is for learners to experience "natural" distribution Of of trajectories, P(r), as compared to the "unnatural" distributions which can result when meta-learning is applied. Formally, we can characterize the natural distribution as:
P(T) / (T))dL
(I)
where L is learner, selected random according to fixed distribution P(L). Here, learner is a fully described learning algorithm" and Pa (T] is the distribution over trajectories that results from running the algorithm in an environment p. Importantly, L is sampled from P(L). instead of being chosen via meta-learning. To provide learners with distribution approximating P(T), context swapping relies on training a population of N learners (I1, ..., IN) in parallel. Each learner inhabits one of N copies EINN= = EE ..., EN} } of the same environment μ.. The E share the same initial state distribution and time-step, but may be in different states on any particular time-step.
The technique of context swapping consists in shuffling the learners through the different copies of the environment, so which copy a given learner inhabits can change at any (or every) time-step. In this work, we use deterministic permutation of learners against environment copies, so that learner Lj acts in copy E, on time-steps t and only if j = (i + t) mod N. When N is larger than the interval of the OL optimizer, each learner will inhabit each copy for at most a single time-step before
2 Note that this concept is different from the reference distribution mentioned in 3.1 the "natural" (non-meta- learned) distribution might still exhibit SIDS, and does so in both of the environments we introduce. For example, T. might completely specify a deep learning algorithm including the choice of initial parameters, and P[L.) might be the distribution induced by the randomness of the initialization.
Note that the learners consist entirely of software; any hardware (e.g. robot body) would be considered part of an environment, typical in reinforcement learning (Sutton & Barto 1998).
4

---

Under review as a conference paper at ICLR 2020
an OL step is applied. This removes the incentive for learners to manipulate the future states they encounter, although they may still have incentives to influence each others' future states. Under the assumption that different copies of the environment do not influence each other, this technique can address HIDS in practice, as we show in Sec. 4.1.1
EXPERIMENTS
To clearly introduce the concepts of SIDS and HIDS, we opt for simple illustrative environ- ments. Code for our experiments is available https: http:////nnoooouso/oo 4open. science/ d09 (09-5e3beed77lllbbeele
k -asi5aa4224444777/7
In Section we introduce a "unit test" for HIDS. Our primary goal with this unit test is for the reader to walk away with a crisp understanding of HIDS. Put simply, our experiments show that you can have a learner which behaves intended, and just by using meta-learning (e.g. PBT), and without changing the performance metric (e.g. loss rewards), the learner's s objective can change completely, leading to unintended behavior. On the practical side, the unit test can be used to diagnose and compare learning algorithms. We show that context swapping is effective mitigation technique in this environment.
In Section 4.2 model a content recommendation system. The goal of these experiments is to provide a practical understanding of different types of SIDS (concept shift and covariate shift), and to demonstrate how HIDS could create issues for real-world recommender systems. We emphasize that SIDS takes place this environment by construction. The point of our experiments is that meta-learning can increase the rate and/or extent of SIDS. Context swapping is not effective in this environment, highlighting the need for alternatives.
4.1 HIDS UNIT TEST
We test whether algorithms reveal hidden incentives using a version of the prisoner's 's dilemma (Prisner
2014) where agent plays each round against its past self. The reward function is presented in Table 1 An agent in this environment has a long-term, non-myopic incentive for cooperation (with its future self), but surrent-time-ste myopic incentive for defection (from its future self). The unit test evaluates whether a learning algorithm reveals the non-myopic incentive even when the agent is meant optimize for the present reward only (i.e. uses discount rate 7 = 0). Naively, we' ' expect the non-myopic incentive to be hidden from the agent in this case, and for the agent to consistently de fect; learning algorithms that do so pass the test. But some learning algorithms also fail the unit test, revealing the incentive for the agent to cooperate with its future self. While aiming for myopic behavior may seem odd, myopic learners have no incentives to cause distributional shift, since can only improve future performance. And while making learners myopic is an extreme approach to managing HIDS, doing so proves to be non-trivial.
Table 1: Rewards for the HIDS unit test. Note that the myopic defect action always increases reward at the current time-step, but decreases reward at the next time-step the incentive hidden from the point of view of myopic learner. A supposedly myopic learner 'fails' the unit test if the incentive to cooperate revealed, i.e. if we see more coope rate actions than de fect.
at defect
04 cooperate
8 11-1 defect
1/2
8; 0:-1 cooperate
1/2
Formally, this environment is not a 2x2 game (like the original prisoner's dilemma); it a partially observable Markov Decision Process (POMDP) (Aström 1965
Kaelbling et al. 1998):
8: 0r-1
0, O
a; E (defect, cooperate]
P(s a.) 0:
R(s,, a:) 1(s cooperate) + B1 cooperate) 1/2
where I is indicator function, and 3 - - is parameter controlling the alignment of incentives (see Appendix for an exploration of different 3 values.). The initial state is sampled as so ~ U U(de fect, cooperate)
5

---

Under review as a conference paper at ICLR 2020
4.1.1 HIDS UNIT TEST EXPERIMENTAL RESULTS AND DISCUSSION
We first show that agents trained with PBT fail the unit test more often when compared with baseline agents that do not use meta-learning. We use REINFORCE (Williams) 1992) with discount factor y = as the IL optimizer for these experiments. Policies are represented by single real- valued parameter 8 (initialized
(O 1)) passed through a sigmoid whose output represents P(a, = defect). PBT (with default settings, see Section 2.2) is used to tune the learning rate, with reward on the final time-step of the interval as the performance measure for PBT. We initialize the learning rate log-uni log-uniformly between 0.01 and 1.0 for all experiments (whether using PBT or not). We expect and confirm that the following two factors lead to higher rates of failure (cooperation):
Shorter intervals: These give the OL more opportunities influence the population.
2. Larger populations: These make outliers with exceptional non-myopic performance more likely, and OL makes them likely to survive and propagate.
The baseline (no PBT) agents pass the unit test: P(cooperate) (averaged over agents) close to 0% see blue curves in Figure 2 However, despite the disincentive for cooperation and the myopic inner loop, agents trained with PBT and large populations fail the unit test: P(cooperate) is around 90%- see the top right subplot of Figure 2
Furthermore, we verify that context swapping significantly mitigates the effect of HIDS, decreasing undesirable cooperate behaviour to near-baseline levels see bottom rows of Figure This effect can be explained as follows: Because context swapping transfers the benefits of a learner's action to the next learner to inhabit that environment, it increases that learner's fitness, and thereby reduces the relative fitness (as evaluated by PBT's EXPLOIT step) of the non-myopic coope r rate behaviour. We observe some interesting exceptions with the combination of small populations and short PBT intervals. Although context swapping still significantly decreases the effect of HIDS, non-myopic cooperate e behaviour is observed much 20% of the time (for #learners110, T 1; see bettom-left plot).
We also observe that PBT reveals HIDS even when T = We provide a detailed explanation for how this might happen in Appendix 3.1.2] But we also note that for 7' = 1, the explanation that PBT operates on longer time horizon than the inner loop does not apply, making especially surprising that HIDS are revealed. Thus we hypothesize that there are at least 2 mechanisms by which PBT is revealing HIDS: (1) optimizing over a longer time-scale, and (2) picking up on the correlation between an agent's 's current policy and the underlying state. Mechanism (2) can be explained informally as reasoning as: "If I'm cooperating, then was probably cooperating on the last time-step as well, so my reward should be higher". As support for these hypotheses, run control experiments identifying two algorithms (each sharing only one of these properties) that can fail the unit test (although context swapping remains effective):
Optimizing over a longer time-scale: replacing PBT with REINFORCE as an outer-loop optimizer.The The outer-loop optimizes the parameters to maximize the summed reward of the last T time-steps. As with PBT, we observe non-myopic behavior, but now only when T >> This supports our hypothesis that the exploitation of HIDS is due not to PBT in particular, but just to the introduction of sufficiently powerful meta-learning. See Figure 2 for results.
Exploiting correlation: Q-learning with 7 0 an E = 0. 0.1-greedy behavior policy and no meta-learning If either state was equally likely, the Q-values would be the average of the values each column in Table I, so the estimated ((defect) would be larger. But the c-greedy policy correlates states and actions, so the top-left and bottom-right entries carry more weight in the estimates, sometimes causing Q(defect) R (ccooperate) and persistent nonmyopic behavior. See Figure 3 for results, Appendix 3.1 for more results, and Appendix 3.1.3 for important experimental details.

---

Under review as a conference paper at ICLR 2020
#agents-10
#agents-100
#agents-
1.0
0.8
0.4
0.2
PBT
PBT interval
1.0
PBT interval
0.8
CTV swapping
interval 100
0.6
0.4
0.2
500
750
1000
250
500
750
1000
250
500
750
1000
tieessepp
time-step
(a) Outer-loop (OL) optimizer Population- Based Training (PBT)
#ogents-10
#egerts
#ogents
1000
CTM S180gigg
OL interval =
interval =
1.00
0.00
8me-step
(b) Outer-loop (OL) optimizer is REINFORCE
Average level non-myopic cooperate behavior observed the unit test for HIDS, with and without two different meta-learning algorithms (PBT and REINFORCE). Lower is better, since the goal is for non-myopic incentives to remain hidden. Despite making the inner loop fully myopic (7 0), both outer-loop optimizers reveal HIDS, however, leading agents to choose the ecoperate e action (top rows of (a) and (b)). Environmentt-wapping significantly mitigates HIDS (bottom rows of (a) and (b)).
Figure 2:
0.0
0.0
0.0
Q-values
0.00
-0.55
-0.5
0.5
-0.4
0.4
-0.50
Picooperate)
1.0
1.0
1.0
1.00
0.9
0.9
0.95
0.8
0.8
2000
2000
2000
0
2000
0
2000
time-step
time-step
time-step
time-step
time-step
O(defect)
O(cooperate)
#cooperate/time-stee
Figure 3: Q-learning sometimes fails the unit test; empirical p (cooperate) stays around 80-90% in of experiments (bottom row). Each column represents independent experiment. Q-values for the cooperate and de fect actions stay tightly coupled in the failure cases (col. 1,2,5), while in the cases passing the unit test (col. 3,4) the Q-value of coope reteraat rate is driven down over time.
7

---

Under review as a conference paper at ICLR 2020
4.2 HIDS IN CONTENT RECOMMENDATION
We now present a toy environment for modeling content recommendation of news articles, which includes the potential for SIDS by incorporating the mechanisms mentioned in Sec. [2.T] discussed as contributing factors to the problems of fake news and filter bubbles. Specifically, the environment assumes that presenting an article to a user can influence (1) their interest in similar articles, and (2) their propensity to the recommendation service. These correspond to modeling self-induced concept shift of users, and self-induced covariate shift of the user base, respectively (see Sec. (21). The environment is designed be as simple possible while incorporating both of these effects.
This environment includes the following components, which change over (discrete) time: User type: x², Article type: y', User interests: W (propensity for users of each type click articles of each type), and User loyalty: g' (propensity for users of each type to use the platform). At each time step a user xf is sampled from categorical distribution, based on the loyalty of the different user types. The recommendation system (a classifier) selects which type of article to present in the top position, and finally the user 'clicks' an article y², according to their interests.
User loyalty for user type I' undergoes covariate shift: in accordance with the self-selection effect. g" increases or decreases proportionally to that user type's interest in the top article. The interests of user type x² (represented by a column of W*) also change, undergoing concept shift; in accordance with the illusory truth effect, their interest in the topic of the top article (as chosen by the recommender system) always increases. The update rates of g', W are specified by on, 022-
Formally, this environment is similar a POMDP R, i.e. POMDP with no reward function, also known as world model (Armstrong & O' O'Rourke, 2017; Hadfield-Menell et al., 2017); the difference is that the learner observes the input (opre) before acting and only observes the target (ooa)) after acting. The states, observations, and actions given below. For further details on this environment, including the state transition function, see Appendix 3.2.T
s² (g' W ", opres a', opost (x' o0 .y')
4.2.1 CONTENT RECOMMENDATION EXPERIMENTAL RESULTS AND DISCUSSION
Our recommender system is I-layer MLP trained with SGD -GD-mmmen Actions are sampled from the MLP's predictive distribution. For PBT, we use = 10 and 20 agents, and use accuracy to evaluate performance. We run 20 trials, and match random seeds for trials with and without PBT. See Appendix 3.2.2 for full experimental details.
We find that PBT yields significant improvements in training time and accuracy, but also greater distributional shift; see Figure 4 User base and user interests both change faster with PBT, and in particular user interests change more overall. We observe that the distributions over user types typically saturate (to a single user type) after a few hundred time-steps (Figure Figure 4 Right). We run long enough to reach such states, to demonstrate that the increase in SIDS from PBT is not transitory. The environment has a number of free parameters, and our results are qualitatively consistent long as (1) the initial user distribution approximately uniform, and (2) the covariate shift rate (a1) is faster than the concept shift rate (a2)) See Appendix 3.2.4 for details.
We measure concept shift (change in P(y|x) as the cosine distance between each user types' initial and current interest vectors. And we measure covariate shift (change in P(x)) as the KL-divergence between the current and initial user distributions, parametrized by g' and g', respectively. In Figure 5) we plot concept shift and covariate shift as function of accuracy. We observe that for both types of SIDS, low levels of accuracy PBT actually causes less shift than occur in baseline agents; HIDS are only observed for accuracies above 60%. This suggests that only relatively strong performers are able to pick up on the HIDS revealed by PBT. See Figure
8

---

Under review as a conference paper at ICLR 2020
Accurocy of click prediction
Chonge in PLY[X] (user interesial
Change in "(XI (user base)
0ottee difancce Pett orgira Intereets
from aitmouttn
2.D
0.04
0-3
0.6-
0.-
0.2
0.01
830
1200
1600
920
800
1200
1600
2000
800 1200 1600 Lmmestap
2000
PDT
-
PST
Figure 4: Content recommendation experiments. Left: using Population Based Training (PBT) increases accuracy of predictions faster, leads to a faster and larger drift in users' interests, P(y|x), (Center); as well as the distribution of users, P(x), (Right). Shading shows std error over 20 runs.
10000
4.00
3.07
3.300
Figure 5: Amount of self-induced covariate shift (left) and self-induced concept shift (right) as a function of performance (accuracy) averaged over all trials, learners, and time-steps. Only relatively strong learners (those which achieve accuracy > 60%) exhibit HIDS.
RELATED WORK
SIDS in practice: We introduce the term SIDS, but we are far from the first to study such problems. Caruana et al.] (2015) provide example of asthmatic patients having lower predicted risk of pneumonia. Treating asthmatics with pneumonia less aggressively on this basis would be an example of harmful SIDS; the reason they had lower pneumonia risk was because they had received more aggressive lung-related care already. [Schulam & Saria (2017) note that such predictive models are commonly used to inform decision- this patient fare with less aggressive treatment") to avoid making such (potentially) self-refuting predictions. While their goal make accurate predictions in the presence of SIDS, our goal is to identify and manage incentives for SIDS, Environments with agents that that react to a learner (such as adversaries) naturally produce SIDS. Goodfellow (2019) argues that adversarial defenses that do not account for distributional shift are critically flawed.
and propose modeling counterfactuals (e.g. "how would
Non-ii.d bandits: Contextual bandits (Wang et al., 2005 Langford & Zhang 2008) are frequently discussed approach to content recommendation (Li et al., 2010). While bandit algorithms typically make the i.i.d. assumption, counter-examples exist (Gheshlaghi Azar et al. 2014 Shah al. 2018); most famously, adversarial bandits (Auer al.) 1995). Closest (2018), who consider self-induced covariate shift in the context of multi-armed bandits. Our task in Sec. is similar their problem statement, but more general in that include user features, thus disentangling covariate shift and concept shift. Our motivation is also different: (Shah et al. (0018) seek to exploit SIDS, whereas we aim avoid hidden incentives for SIDS.
work Sh/h et al.
Understanding and managing the incentives of learners is also a focus of Armstrong & O' 'Rourke (2017) 7): [veritt (018);; Everitt et al. (2019); Cohen et al. (2019). Emergent incentives to influence the world (such as HIDS) are at the heart of many concerns about the safety of advanced AI systems (Omohundro/ 2008 Bostrom 2014). Yet it is unclear if or when machine learning systems might pursue such "instrumental goals' in practice. Indeed, Drexler (2019) argues that machine learning should and typically does use time- and ssource-bbunded problem statements, making dangerous instrumental goals less likely to emerge. The same idea underlies
Safety and incentives:

---

Under review as a conference paper at ICLR 2020
several more concrete approaches to building safe superintelligent Al systems: myopic reinforcement learning (Leike et al., 2018; Knox & Stone 2008) Cohen et 20199 and its application in iterated
amplification (Christiano et a 2018/Cotre (017) and question answering systems (Everitt et al. 2019; Armstrong & O' Rorke| 2017). Managing HIDS seems critically important for the safety of these approaches: they rely on enforcing myopia, which our experiments show is not straightforward.
HIDS and meta-learning: As far as we know, our work is the first to consider the problem of HIDS, or its relation to meta-learning. A few previous works have some relevance or resemblance. Rabinowitz (2019) documents qualitative differences in learning behavior when meta-learning is applied. MacKay et al. (2019) and Lorraine & Duvenaud (2018) view meta-learning as a bilevel optimization problem, with the inner loop playing a best-response to the outer loop. In our work, the inner loop is unable to achieve such best-response behavior; the outer loop is too powerful (see Fig. 2) Finally, Sutton et al. (2007) note that meta-learning can change learning behavior in way that improves performance by preventing convergence of the inner loop. Their goal of improving performance by "tracking" local characteristics of the environment is orthogonal to our goal of managing incentives to control such local characteristics.
DISCUSSION AND CONCLUSION
We have identified the phenomenon of self-induced distributional shift (SIDS), and the problems that can arise when there are hidden incentives for algorithms to induce distributional shift (HIDS). Our work highlights the interdisciplinary nature of issues with real-world deployment of MI. systems we show how HIDS could play a role in important technosocial issues like filter bubbles and the propagation of fake news. There are number of potential implications for our work:
1. When HIDS a concern, our methodology and environments can be used to help diagnose whether and to what extent the final performance/behvion of a learner is due to SIDS and/or incentives for SIDS, i.e. quantify their influence on that learner.
2. Comparing this quantitative analysis for different algorithms could help us understand which features of algorithms affect their propensity to reveal HIDS, and aid in the development of safer and more robust algorithms.
Characterizing and identifying HIDS in these tests is first step to analyzing and mitigating other (problematic) incentives, as well as to developing theoretical understanding of incentives.
Broadly speaking, our work emphasizes that the choice of machine learning algorithm plays an important role in specification, independently of the choice performance metric. A learner can use SIDS increase performance according to the intended performance metric, and yet still behave in an undesirable way, if we did not intend the learner to improve performance by that method. In other words, performance metrics are incomplete specifications: they only specify our goals or ends, while our choice of learning algorithm plays a role in specifying the means by which we intend an learner to achieve those ends. With increasing deployment of ML algorithms in daily life, we believe that (I) understanding incentives and (2) specifying desired/allowed means of improving performance are important avenues of future work ensure fair, robust, and safe outcomes.
REFERENCES
Hunt Allcott and Matthew Gentzkow. Social media and fake news in the 2016 election. Journal of Economic Perspectives, 31 May 2017.
Michelle A. Amazeen and Bartosz W. Wojdynski. Reducing native advertising deception: Revisiting the antecedents and consequences of persuasion knowledge in digital news contexts. Mass Communication and Society, 0(0):1-26, 2018.
Marein Andrychowicz, Misha Denil, Sergio Gómez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Neural Information Processing Systems, 2016.
Stuart Armstrong and Xavier O' Rorke. Good and safe uses of AI oracles. ArXiv preprint, November 2017.
Stuart Armstrong and Xavier O' Rourke. Indifference methods for managing agent rewards. Technical report, Future of Humanity Institute, 2017.
10

---

Under review as a conference paper at ICLR 2020
K. J. Åström. Optimal control of Markov Processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10:174-205, January 1965.
P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In Foundations of Computer Science, 1995.
Eytan Bakshy, Solomon Messing, and Lada A. Adamic. Exposure to ideologically diverse news and opinion on Facebook. Science, 348(6239):1130-1132, 2015. ISSN 0036-8075. doi: 10.1 10.1126/cciecceaaae 160. URL http: http://ciencccc swwenceeeaaaaeeeee wttpooooeeoo/////eo/.oo/..
6239/1130
Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Inc., New York, NY, USA, 1st edition, 2014.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In International Conference on Knowledge Discovery and Data Mining, 2015.
Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. ArXiv preprint, 2018.
Michael K. Cohen, Elliot Catt, and Marcus Hutter. A strongly asymptotically optimal agent in general environments. Proceedings of the Twenty-Eighth International Joint Conference on Arlificial Intelligence, Aug 2019. doi: (0.24963/jcaa.20199333 URL http:htpp:// //tp:://xx.o doi org
10.
Ajeya Cotra. Iterated distillation and amplification. Technical report, AI Alignment, 2017.
Dominic DiFranzo and Kristine Gloria-Garcia. Filter bubbles and fake news. XRDS, 23(3):32-35, April 2017. ISSN 1528-4972. doi: 105334553 URL http::/do// www.arm...o......... org/10
1145/3055153
K. Eric Drexler. Reframing superintelligence: Comprehensive Al services as general intelligence. Technical report, Future of Humanity Institute. 2019.
Mostafa M. El-Bermawy. Your echo chamber is destroying democracy, 2016. URL https 7/ w/www.c wi red.
Tom Everitt. Towards Safe Artificial General Intelligence. PhD thesis, Australian National University, 2018.
Tom Everitt, Pedro Ortega, Elizabeth Barnes, and Shane Legg. Understanding agent incentives using causal influence diagrams, part i: Single action settings, 2019.
Lisa K. Fazio, Nadia M. Brashier, B. Keith Payne, and Elizabeth J. Marsh. Knowledge does not protect against illusory truth. Journal of Experimental Psychology: General, 14465))993-1002, 10 2015. ISSN 0096-3445. doi: 10. 10.10077xge00000098.
Chelsea Finn, Pieter Abbecl, and Sergey Levine Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, 2017.
Seth Flaxman and Sharad Goel. Filter bubbles, echo chambers, and online news consumption. Public Opinion Quarterly, 2015.
Conor Friedersorf. Youtube extremism and the long tail: Unlimited selection is revealing ugly truths about what some americans want in their politics. 2018.
Oguzhan Gencoglu, Mark van Gils, Esin Guldogan, Chamin Morikawa, Mehmet Süzen, Mathias Gruber, Jussi Leinonen, and Heikki Huttunen. HARK side of deep learning from grad student descent automated machine learning. ArXiv preprint, 2019.
Mohammad Gheshlaghi Azar, Alessandro Lazaric, and Emma Brunskill. Online stochastic optimiza- tion under correlated bandit feedback. ArXiv preprint, 2014.
11

---

Under review as a conference paper at ICLR 2020
Dong Gong, Zhen Zhang, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, and Yanning Zhang Learning an optimizer for image deconvolution. ArXiv preprint, 2018.
lan J. Goodfellow. research agenda: Dynamic models to defend against correlated attacks. ArXiv preprint, 2019.
Jacob Groshek and Karolina KochMiicalska. Hclping populism win? Social media use, filter bubbles, and support for populist presidential candidates in the 2016 us election campaign. Information, Communication & Society, 20(9): 1407, 2017. doi: 10.1080/1369 30000///6111100001111 (0....X21..1229992 URL httpp::/dbi..rrg/ 1ttp://1...000011111 18x. 2017. 1329334 2017.1323334
Dylan Hadnilld-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. In Neural Information Processing Systems, 2017.
M. Jaderberg, V. Dalibard, S. Osindero, W. M. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, I. Dunning, K. Simonyan, C. Fernando, and K. Kavukcuoglu. Population Based Training of Neural Networks. ArXiv preprint, 2017.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, (01(12))99-13444 May 1998. ISSN 0004-3702.
A. Kalousis. Model selection via meta-learning: A comparative study. In IEEE International Conference on Tools with Artificial Intelligence, 2000.
Varol Kayhan, Confirmation bias: Roles of search engines and search contexts. In International Conference Information Systems, 2015.
W. Bradley Knox and Peter Stone. TAMER: Training Agent Manually via Evaluative Reinforce- ment. In IEEE 7th International Conference on Development and Learning, 2008.
John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Neural Information Processing Systems, 2008
Ed. Lee Howell. Digital
wildfires in hypercomnected world. In Lee URL
Howell (ed.), Global Risks 2013. World Economic Forum, 2013.
http::/
wwforum.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq. Laurent Orseau, and Shane Legg. AI safety gridworlds. Technical report, DeepMind Safety Research, 2017.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction. Technical report, DeepMind Safety Research, 2018.
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In International Conference on World Wide Web, 2010.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypernet- works. ArXiv preprint, 2018.
D. D. Luxton, D. June, and J. M. Fairall. Social media and suicide: public health perspective. American journal of public health, 102(2):195-20) 2012.
Matthew MacKay, Paul Vicol, Jonathan Lorraine, David Duvenaud, and Roger Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. ArXiv preprint, 2019.
Merrian-Webster. The real story of fake news, 2017. URL http://////www.rrrraaeeeeessss
com/wordss
news
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Learning unsupervised
learning rules. In International Conference Learning Representations, 2019.
12

---

Under review as a conference paper at ICLR 2020
Paul Mihailidis and Samantha Viotty. Spreadable spectacle in digital culture: Civic expression, fake news, and the role of media literacies in "post-fact" society. American Behavioural Scientist, 2017.
Jose G. Moreno-Torres Troy Raeder, RocíO Alaiz-RodriGuez, Nitesh V. Chawla, and Francisco Herrera. A unifying view dataset shift in classification. Pattern Recognition, 45(1):521-530, January 2012.
Tien T. Nguyen. Pik-Mai Hui, F. Maxwell Harper, Loren Terveen, and Joseph A. Konstan. Exploring the filter bubble: The effect of using recommender systems on content diversity. In Proceedings World Wide Web, WWW '14, pp. 677-686, New York, doi: 10.1145/2566466556811
of the 23rd International Conference NY, USA, 2014. ACM. ISBN 978- -454--------
URL
http://doi acm. erq/10. 1tt.://..1145226666.688 2568012
Safyy Umoja Noble. Algorithms of Oppression: How Search Engines Reinforce Racism. NYC Press, 2018.
Stephen M. Omohundro. The basic AI drives, In Conference on Artificial General Intelligence, 2008.
Pedro A. Ortega and Vishal Maini. Building safe artificial intelligence: specification, robustness, and assurance, 2018.
Eli Pariser. The Filter Bubble: What the Internet Is Hiding from You. The Penguin Group. 2011.
Gordon Pennycook, Tyrone D Cannon, and David G. Rand. Prior exposure increases perceived accuracy of fake news. Journal of Experimental Psychology (forthcoming), 2019.
Erich Prisner. Game Theory Through Examples. Mathematical Association of America, 2014.
Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence. Dataset Shift in Machine Learning. The MIT Press, 2009.
Neil C. Rabinowitz. Meta-learners' learning dynamics unlike learners'. ArXiv preprint, 2019.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-leaming for semi-supervised few-shot classification. ArXiv preprint, 2018
Manoel Horta Ribeiro, Raphael Ottoni, Robert West, Virgilio A. F. Almeida, and Wagner Meira. Auditing radicalization pathways on youtube, 2019.
David Robson. The myth of the online echo chamber, 2018. URL http://ttp/yy bbc com/ Eaturre/story/2218444 thettttthhhhehhhhhhh--ee-- -he www.einiineeecccccacaaaaeeeeeeee
Kevin Roose. The making of youtube radical. 2019.
Peter Schulam and Suchi Saria. Reliable decision support using counterfactual models. In Neural Information Processing Systems, 2017.
Virag Shah, Jose Blanchet, and Ramesh Johari. Bandit learning with positive externalities. Neural Information Processing Systems, 2018.
Chengeheng Shao, Giovanni Luca Ciampaglia, Onur Varol, Kai- Lui-Cheng Yang, Alessandro Flammini, and Filippo Menczer. The spread of low-credibility content by social bots. Nature Communications, ((4787) 2018.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. In Neural Information Processing Systems, 2012.
Richard S Sutton and Andrew G Barto. Introduction to Reinforcement Learning. MIT Press, 1998.
Richard S Sutton, Anna Koop, and David Silver.
On the role of tracking in stationary environments. In International conference on Machine learning, 2007.
Techopedia. Filter bubble, 2018. URL https: /tw:://wwwww.. htttp::/wwww.eechoce.. httpehhpppeaaa.ccmeeieiiiiii...e wwt.://iiiieeeciiooonoommmoma.oe 28556/
13

---

Under review as a conference paper at ICLR 2020
Chih-Chun Wang, Sanjeev R Kulkarni, and H Vincent Poor. Bandit problems with side observations. IEEE Transactions on Automatic Control, 50(3):338- 335, 2005.
Wikipedia contributors. Confirmation bias - Wikipedia, the free encyclopedia, 2018. URL
https://en. www.wedia.......mmmmmmmm........ org//w.rrwww... w i n http:/innxx.pppppp.epeeepp www.:pppeeeeeeeiiiiiii.mmme.ei wont ion
ion_biasa biass
oldid=875026725 [Online; accessed 20-January-2019).
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, pp. 229 256,, 1992.
14

---

Under review as a conference paper at ICLR 2020
APPENDICES
CONTENT RECOMMENDATION IN THE WILD
Filter bubbles, the spread of fake news, and other techno-social issues are widely reported to be responsible for the rise of populism (Groshek & Koo- Michalsaa 2017), increase in racism and prejudice against immigrants and refugees (Noble] 2018), increase in social isolation and suicide (Luxton al., 2012), and, particularly with reference to the 2016 US elections,
decried as threatening the foundations of democracy [EI-Bernawy] 2016)). Even in 2013, well before the 2016 American elections, World Economic Forum report identified these problems global crisis
(ee Howell 2013).
We focus on two related issues in which content recommendation algorithms play a role: fake news and filter bubbles.
1.1 FAKE NEWS
Fake news (also called false news junk news) is extreme version of yellow journalism, pro- paganda, or clickbait, in which media that is ostensibly providing information focuses on being eye-catching or appealing, at the expense of the quality of research and exposition of factual informa- tion. Fake news is distinguished by being specifically and deliberately created to spread falsehoods or misinformation (Merriam- -eebstrr 2017; Mihailidis & Viotty 2017).
Why does fake news spread? It may at first seem the solution is simply to educate people about the truth, but research tells us the problem is more multifaceted and insidious, due to a combination of related biases and cognitive effects including confirmation bias (people are more likely to believe things that fit with their existing beliefs), priming (exposure to information unconsciously influences the processing of subsequent information, i.e. seeing something in credible context makes things seem more credible) and the illusory truth effect (i.e. people are more likely to believe something simply if they are told it is true).
Allcott & Gentzkow (017)) track about 150 fake news stories during the 2016 US election, and find the average American adult saw 1-2 fake news stories, just over half believed the story was true, and likelihood of believing fake news increased with ideological segregation (polarization) of their social media. [Shao al.) (018)) examine the role of social bots in spreading fake news by analyzing 14 million Twitter messages. They find that bots are far more likely than humans to spread misinformation, and that success of fake news story (in terms of human retweets) was heavily dependent on whether bots had shared the story.
Pennycook et al. (2019) examine the role of the illusory truth effect in fake news. They find that even a single exposure to news story makes people more likely to believe that it is true, and repeat viewings increase this likelihood. They find that this is not true for extremely implausible statements (e.g. "the world is perfect cube"), but that "only a small degree of potential plausibility is sufficient for repetition to increase perceived accuracy" of the story. The situation further complicated by peoples' inability to distinguish promoted content from real news Amazeen & Wojdynski] (2018) find that fewer than 1/10 people were able to tell when content was an advertisement, even when was explicitly labelled as such. Similarly, Fazio et al. (2015) find that repeated exposure to incorrect trivia make people more likely to believe it, even when they are later able to identify the trivia as incorrect.
1.2 FILTER BUBBLES
Filter bubbles, a term coined and popularized by Pariser (0011) are created by positive or negative feedback loops which encourage users groups of users towards increasing within-group similarity, while driving up between setween-group dissimilarity. The curation of this echo chamber is called self- selection (people favours what Techopedial (2018) calls intellectual isolation. In the context of social and political opinions, this is often called the polarization effect (Wikipedia contributors. 2018).
more likely look for select things that fit their existing preferences), and
15

---

Under review as a conference paper at ICLR 2020
Filter bubbles can be encouraged by algorithms in two main ways. The first is the most commonly described: simply by showing content that is similar to what a user has already searched for, search or recommender systems create a positive feedback loop of increasingly-sinm
content (Pariser 2011 Kayhan 2015). The second way is similar but opposite- if the predictions of an algorithm are good for certain group of people, but had for others, the algorithm can do better its metrics by driving hard-to-predict users away. Then new users to the site will either be turned off' entirely, or see an artificially homogenous community of like-minded peers, a phenomena (Shah et al. (2018) call positive externalities.
In study of 50,000 US-based internet users, Flaxman & Goel (2015) find that two things increase with social media and search engine use: (1) exposure of an individual to opposing or different viewpoints, and (2) mean ideological distance between users. Many studies eite the first result as evidence of the benefits of internet and social media (Robson, 2018 Bakshy et al., 2015), but the correlation of exposure with ideological distances demonstrates that exposure is not enough, and might even be counterproductive.
Facebook's own study on filter bubbles results show that the impact of the news feed algorithm on filter bubble "size" (a measure of homogeneity of posts relative to a baseline) is almost as large as the impact of friend group composition (Bakshy et al. 2013). Kayhan (2015) specifically study the role of search engines in confirmation bias, and find that search context and the similarity of results in search engine results both reinforce existing biases and increase the likelihood of future biased searches. Nguyen et al. (2014) similarly study the effect of recommender systems individual users' content diversity, and find that the set of options recommended narrows over time.
Filter bubbles create an ideal environment for the spread of fake news: they increase the likelihood of repeat viewings of similar content, and because of the illusory truth effect, that content is more likely to be believed and shared (Pennycook et al.) 2019 DiFranzo Gloria- Mloria-Gareia, 2017; Pariser 2011). We are not claiming that HIDS are entirely or even mostly responsible for these problems, but we do note that they can play a role that is worth addressing.
FORMAL DEFINITIONS
Here we formalize the concepts of learning scenario, operational context, and problem statement for maximal clarity and to highlight issues of applying machine learning in practice. A learning scenario, consists of a learning algorithm, L, operational context, w, and performance metric, The performance metric is quantification of learners' performance, used during testing, validation, and sometimes training, to evaluate and compare learners. The operational context is the weal-world setting where learner operates. It includes things like the training data or reinforcement learning environment. Significantly, also includes external factors, such as human users and the computer hardware running the learning algorithm, which may potentially influence the data states the learner encounters. The learning algorithm instantiates a (potentially stochastic) mapping:
L (w, L) O
(2)
where O is the output of learning, including things like a learned model, learning curves, and/or complete log of any computations or data processed during learning.
problem statement, S, is model of an operational context, used by humans analyze properties of learning scenarios. Like all models, problem statements make simplifying assumptions, such as assuming i.i.d. inputs in the presence of distributional shift. Researchers often design a learning algorithm with a specific problem statement, 3 iintended, in mind, and only evaluate it in operational contexts that are carefully controlled to match & inttenddd On the other hand, practitioners often deploy learning algorithms in less controlled operational contexts that are not faithful to intended Our work employs and advocates for an empirical methodology of testing a learning algorithm designed for & iintended in a problem statement @realistic which seems more realistic (but may still fail to capture important aspects of an operational context), in order to detect possible failure modes and develop mitigation strategies.
16

---

Under review as a conference paper at ICLR 2020
3 EXTRA EXPERIMENTS AND REPRODUCIBILITY DETAILS
3.1 HIDS UNIT TEST
3.1.1
ALIGNMENT INCENTIVES EXPLORATION
This section presents an exploration of the parameter B, which controls the alignment incentives in the HIDS unit test.
To clarify the interpretation of experiments, we distinguish between environments in which myopic (defect) nonmyopic (cooperate) incentives opposed, orthogonal, compatible. Note that in this unit test myopic behaviour (defection) is what we want to see.
Incentive-opposed: Optimal myopic behavior is incompatible with optimal nonmyopic behav- ior (classic prisoner's 's dilemma; these experiments in the main paper).
2. Incentive-orthogonall: Optimal myopic behavior may or may not be optimal nonmyopic behavior.
Inoeative Optimal myopic behavior is necessarily also optimal nonmyopic behav- ior.
Table 2: 3 controls the extent to which myopic and nonmyopic incentives are aligned. Cooperating
8
Environment
0
incentive-opposed incentive-orthogonal
yields less reward the current time-step (myopically detrimental)
does not affect the current reward (myopically indifferent) yields more reward the current time-step (myopically beneficial)
incentive-compatible
We focused on incentive-opposed environment (8 1122) the main paper in order to demonstrate that HIDS can be powerful enough to change the behavior of the system in undesirable way. Here we also explore incentive-compatbbee and incentive-orthooonall environments because they provide useful baselines, helping us distinguish systematic bias towards nonmyopic behavior from other reasons (such as randomness or optimization issues) for behavior that does not follow myopically optimal policy.
17

---

Under review as a conference paper at ICLR 2020
#agents=10
#agents=100
#agents=1000
1.00
Buddeaas AUD
Piverbose)
0.50
0.25
0.00
1.00
Budddess aRM
0.75
0.50
time-step
time-step
time-step
(a) incentive-compatible (3 0.5)
#agents=10 10
#agents=100
#agents=1000
1.00
Buiddews aus
0.75
0.50
0.25
1.00
Fiverbsse)
0.75
Gurddews AUd
0.25
0.00
250
500
750
1000
250
500
750
1000
250
500
750
1000
time-step
time-step
time-step
(b) incentive-orthogonal (3 = 0.0)
#agents=10
Wagents=100
#agents=1000
1.00
bujddoas AUD
0.75
0.50
0.25
0.00
1.00
Suddeass AUSa
0.75
0.50
500
1000
500 time-step
time-step
time-step
(c) decentive-opposeee (8 -0.5)
no PBT
PBT interval =1
PBT interval
PBT interval 100
Figure 6: Average level of non non-myopic (i.e. cooperate) behavior learned by agents in the unit test for HIDS. Despite making the inner loop fully myopic (7 = 0), population- based training (PBT) can cause HIDS, leading agents to choose the cooperate action (top row). Environment--wapping successfully prevents this (bottom row). Columns (from left to right) show results for populations of 10, 100, and 1000 learners. the legend, "interval" refers the interval (I") of PBT (see Sec. 2.2). Sufficiently large populations and short intervals are necessary for PBT to induce nonmyopic behavior.
18

---

Under review as a conference paper at ICLR 2020
3.1.2 WORKING THROUGH DETAILED EXAMPLE FOR PBT WITH T =
To help provide intuition on how (mechanistically) PBT could lead to persistent levels of cooperation. we walk through a simple example (with no inner loop). Consider PBT with 7 = and a population Now
of 5 deterministic agents A..... As playing cooperate and receiving reward of r(4,) suppose A1 suddenly switches to play defect. Then (A11) = 1/2 on the next time-step (while the other agents' reward is still 0), and so PBT's EXPLOIT step will copy A (without loss of generality to A2). On the following step. r( (A2 = 1/2, and r(A1) = -1/2, so PBT will clone A2 to A1, and the cycle repeats. Similar reasoning applies for larger populations, and >
3.1.3 Q-LEARNING EXPERIMENT DETAILS
We show that, under certain conditions, Q-learning can learn (primarily) cooperate, and thus fails the HIDS unit test. We estimate Q-values using the sample-averan method, which is guaranteed to converge in the fully observed, tabular case (Sutton Barto 1998). The agent follows the c-greedy policy with 0.1. In order to achieve this result, additionally start the agent off with one synthetic memory where both state and action are defect and therefor R(defect) hard-code the starting state be cooperate (which normally only happens 50% of the time). Without this kind of an initialization, the agent always learns to defect. However, under these conditions, we find that 10/30 agents learned to play cooperate most of the time, with (ccooperate) and (deffect) both hovering around -0.07, while others learn to always defect, with Q (coopperate) ~ -0.92 and Q (ddefect) z -0.45. context swapping, however, prevents emerging, see Figure 9
-.5, and we
behavior from ever
3.1.4 Q-LEARNING: FURTHER RESULTS
To give a more representative picture of how often O-learning fails the unit test, we run a larger set of experiments with Q-learning, results in Figure It's possible that the failure of Q-learning is not persistent, since we have not proved otherwise, but we did run much longer experiments and still observe persistent failure, Figure 7
0.0
0.0
Rccoppratee Q-vauues
0.0
0
0.5
1.0
1.0
1.0
0.5
0.9
0.8
50000
50000
50000
50000
50000
tie-step
time-step
time-step
time-step
time-step
Q(defect)
O(cooperate)
#cooperate/imee-step
Figure 7: The same experiments as Figures 3 run for 50,000 time-steps instead of 3000, to illustrate the persistence of non-myopic behavior.
19

---

Under review as a conference paper at ICLR 2020
Q-values
0.0
0.0
0.0
0.5
0.5
0.5
Picooperate)
1.0
1.0
1-
1.0
1.0
0.5
0.5
0.5
2500
time-step
time-step
time-step
time-step
time-step
O(defect)
O(cooperate)
#cooperate/time-step
0.0
0.0
0.0
Q-values
-0.5
-0.5
-0.5
00.5
P(cooperate)
0.5
0.5
0.9
2500
2500
2500
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Occooperate)
#cooperate/time-sep
0.0
0.0
0.0
Accoperate) Q-values
-0.5
0.5
0..5
1.0
1.0
1.0
0.5
0.9
0.0
0.0
2500
2500
2500
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Occooperate)
#cooperate/lime-step
Picooperate) Q-values
0.0
0
0.0
-0.5
-0.5
-055
-0.5
1.0
1.0
1.0
0.5
0.9
0.8
2500
time-step
time-step
time-step
Qddffect)
Occooperate)
#cooperate/lime-step
Q-values
0.0
0.0
0.0
-0.5
005
0.5
1.0
P(cooperate)
1.0
1.0
0.5
0.5
0.5
0.5
0.5
2500
2500
2500
0
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Occooperate)
#cooperate/time-ster
Figure 8: More independent experiments with Q-learning, exactly following Figure fails the unit test in total of 10/30 experiments (including those from Figure 3).
Q-learning
20

---

Under review as a conference paper at ICLR 2020
0.5
Q-values
0.0
0.0
0
-0.5
0.5
0.5
-0.5
-1
Plcooperate)
1.0
1.0
1.
1.0
1.0
0.5
0.5
0.5
0.5
0.5
0.0
0.0
2500
2500
2500
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Q(cooperate)
#cooperatejtime-step
0.0
0.0
Q-values
0.0
0.0
-0.5
005
0.5
P(cooperate)
1.0
1.0
0.5
0.5
0.5
0.5
0.5
0.0
2500
2500
2500
0
2500
2500
time-step
time-step
time-step
time-step
time-step
Oldefect)
O(cooperate)
#cooperate/lime-step
0.0
0.0
0.0
Q-values
0.0
0.5
0.5
005
Picooperate)
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
2500
2500
2500
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Q(cooperate)
#cooperate/time-ster
0.0
0.5
Q-values
0.5
0.5
-1
P(cooperate)
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
2500
2500
2500
0
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Q(cooperate)
#cooperate/lime-step
Q-values
-0.5
-1
P(cooperate)
1.0
1.0
0.5
0.5
2500
2500
2500
0
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Occooperate)
#cooperate/time-ster step
Figure 9: More independent experiments with Q-learning, exactly following Figure except also using context swapping. This leads 100% success rate the unit test.
21

---

Under review as a conference paper at ICLR 2020
3.2 CONTENT RECOMMENDATION
3.2.1
ENVIRONMENT DETAILS
The evironment has the following components:
1. User type, z': categorical variable representing different types of users. The content recommender conditions its predictions on the type of the current user.
2. User loyalty, g': the propensity for users of each type to the platform. User x' is sampled from a categorical distribution with parameters given by softmax(g').
3. Article type, y': categorical variable (one-hot encoding) representing the type of article selected by the user.
4. User interests, W': a matrix whose entries W represent the average interest user of type x have in articles of type y.
At each time step different user types), then the recommendation system selects which type of article to present in the top position, and finally, the user selects an article. The goal of the recommendation system is to predict the likelihood that the user would click on each of the available articles, in order to select the one which most interesting to the user.
z' is sampled from a categorical distribution (based on the loyalty of the
User loyalty for x' then changes in accordance with the self-selection effect, increasing or decreasing proportionally their interest in the top article. The interests of user type x² (represented by column of W*) also change; in accordance with the illusory truth effect, their interest in the topic of the top article (as chosen by the recommender system) always increases. Overall, this environment is an extremely crude representation of reality, but it allows us to incorporate both the effects of self-selection (via covariate shift), and the illusory truth effect (via concept shift).
Formally, this environment is similar to POMDP R. i.e. POMDP with no reward function, also known as a world model (Armstrong & O'Ruue 2017: Hadfield-Menn Menell et al. 2017); the difference is that the learner observes the input betore acting and only observes the target after acting. The states, observations, and actions given below.
s'= (g' W W","","" opres a', opast (x', 00 (4)
The state transition function is defined by:
++1 g + a. W2 z1.01
W W 1 x' 1/2
w/+1/2 x¹ 0'
Wt z' + a2;
r04 softmax (g 1
+ softmax w(+h)
atf+1)
Where j' is the top article as chosen by the recommender, and 01, 02 represent the rate of covariate and concept shift (respectively). The update for W merely increases the interest of user type xf in article type g", then normalizes the interests for that user type.
3.2.2 REPRODUCIBILITY DETAILS
For these experiments, the recommendation system is a ReLU-MLP with hidden layer of 100 units, trained via supervised learning with SGD (learning rate 0.01) to predict which article user will select. Actions are sampled from the MLP's predictive distribution. We apply PBT without any hyperparameter selection (this amounts to just doing the EXPLOIT step), and an interval of 10, selecting on accuracy. We use a population of 20 learners (whether applying PBT not), and match random seeds for the trials with and without PBT. We initialize g' and W to be the same across the 20 copies of the environment (i.e. the learners start with the same user population), but these values diverge throughout learning. For the environment, we set the number of user and article types both to 10. Initial user loyalties randomly sampled from (0, 0.03), 0n 0.03, and 02 0.003.
22

---

Under review as a conference paper at ICLR 2020
3.2.3 CONTEXT SWAPPING CONTENT RECOMMENDATION
We believe context swapping not appropriate for the content recommendation environment, since when the environments diverge, optimal behavior may differ across environments. Nevertheless, we ran experiments with it for completeness. The main effect appears be to hamper learning when PBT is not used, see Figure [0, Notably, it does not appear to significantly influence the rate or extent SIDS when combined with PBT.
Accuracy of click prediction
Change in P(Y[X) (user interests)
Change in P(X) (user base)
cosine distance from original interests
2.5
0.05
div. from original distribution
3000006
2.0
0.04
Accuracy
1.5
3.0
0.02
0.01
0.5
0.00
0000
0.0
400
800
1200
2000
800
1200
1600
1200 1600
2000
tme stop
tmme step
stop
no PBT
PBT, env. swapping
no PBT, env. swapping
cosine distance from ortginal interests
Change in P(X) (user base)
Change P(Y[X) (user interests)
KL div. from ongginal distributiin
0,05
2.305
0.04
2.300
0.03
2.295
2.290
0.01
2.285
0.00
0.6
0.8
1.0
0.00
0.25
0.50
0.75
1.00
accuracy
accuracy
PBT
no PBT
PBT, env. swapping
no PBT, env. swapping
Figure 10: ment.
context c doesn't have the desired effect in the content recommendation environ-
3.2.4 EXPLORATION ENVIRONMENT PARAMETERS
In Figure II we examine the effect of the rate-of-change parameters (a1, (22) of content recommendation environment on the results provided in the paper. As noted there, our results are qualitatively consistent so long as (1) the initial user distribution is approximately uniform, and (2) the covariate shift rate (a1) is faster than the concept shift rate (a2). These distributions updated by different mechanisms, and are not directly comparable. Concept shift changes the task more radically, requiring learner change its predictions, rather than just become accurate wider range of inputs. We conjecture that changes in P P(y[æ) must therefore be kept smooth enough for the outer loop have pressure capitalize on HIDS.
23

---

Under review as a conference paper at ICLR 2020
1930
1000
1000
2500
skay
PCYUK) cuser Interests)
FIYDO interests)
302
0.2
310
0.50
2000
Change
Chenge base)
PRT
NIT
P6T
(a) (E) 0.01 02
0.001
(b) 0.1 02
0.001
Accuracy prediction
Accuracy of click prediction
400
000
1320
1000
2000
100
JMM
2009
tmtestop
Chango
0030
3.05
0025
3.00
0.50
1003
600
1220
1000
2000
400
900
1200
2000
2000
imestes
2000
2500
(c) Q11 0.01 0z 0.01
(1
0..1 012
0.01
Accuracy predition
Accuracy prediction
410
800
1756
1600
2000
450
855
1700
1600
2500
Umesspp
Umn stop
Charge PMY10
Charge ga FIY interests)
0.5
0.0
085
0.0
600
1200
1600
2000
400
020
3200
1000
3000
timeste
tine strp
Change
Chenge
800
1200
2000
2000
420
800
1200
1500
2000
(e) [1
0.01, 02 0.1
(0 01
02 =
Figure 11: Content recommendation results for different values of O1,, Q2.
24


Reviews:
[{"review_id": "rJxhKoSaFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study the phenomena of self-introduced distributional shift. They define the term along with the term hidden incentives for distributional shift. The latter describes factors that motivate the learner to change the distribution in order to achieve a higher performance. The authors study both phenomena in two domains (one being a prisoner dilemma and the other a recommender system) and show how meta-learning reveals the hidden incentives for distributional shift. They then propose an approach based on swapping learners between environments to reduce self introduced distributional shift.\n\nIn my opinion this paper should be (weak) rejected for several reasons.\n\n-\tThe paper is poorly written. Some sentences are hard to comprehend, even after repeated reading. It feels hastily written and should be carefully proofread with the help of a proficient English speaker.\n-\tThe very first paragraph is not a helpful example. While the authors provide an example that that illustrates a distributional shift, they don’t describe what the shift is and why that shift in a negative consequence for learning. Therefore, the example is not helpful, but rather confusing.\n-\tIn point 3 of the contributions, the authors state ‘that meta-learning reveals HIDS in these environments’. Is this true for all meta-learning approaches? The current statement overclaims their findings. If the authors decide to keep it, they should provide a description of the types of meta-learning approaches that reveal HIDS and which don’t or evaluate all different meta-learning approaches.\n-\tSection 4.1 is hard to follow. The authors state that the meta-learner is used to tune the learning rate – they fail to clearly explain how exactly the learning rate tuning results in a non-myopic behavior for a myopic learner.\n-\tIn the Q-learning example in Section 4.1, is the effect of \\epsilon considered? The discovery of non-myopic strategies might simply be based on chance. I would like the authors to include an investigation into the effect of this parameter.\n\nGenerally speaking, I appreciate the author’s thoughts on SIDS and how they approach revealing hidden incentives. Although I vote to reject this paper, I strongly encourage the authors to rewrite the paper, address all other issues that are noted during this peer review and resubmit.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xlW-zf9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper discusses concepts of self-induced distributional shift (SIDS) and the hidden incentives when using meta-learning algorithms. It then prescribes a unit-test to check whether there is hidden incentive for distributional shift (HIDS) in the algorithm and proposes to use context swapping to mitigate such phenomenon.\n\nI am not very familiar with this series of research but I am wondering why the paper focuses on meta-learning and its connection to HIDS. Does it use meta-learning as a tool to identify HIDS? But from the description and experiments, it seems the paper is talking about how meta-learning itself leads to HIDS, for example, by comparing different hyper-parameter setting for meta-learning and PBT, it shows the unit-test is failing. So it seems like meta-learning itself leads to distributional shift?\n\nAlso I cannot fully appreciate the utility of this \"unit-test\". It is well known normally an interactive system that can change its inputs have distributional shift. What other information does this \"unit-test\" inform us? Similarly, how does \"context swapping\" mitigate distributional shift? From the experiments, it dumbs the meta-learning algorithms and make it pass the \"unit-test\", but I am not sure what other practical benefits it can bring to improve real systems.\n\nUsually, a reinforcement learning algorithm can meaningfully mitigates the adverse effects of distributional shift by explicitly modeling this interactive process and evaluating rewards with considerations to distributional shift caused by different policies. It is difficult to see how the concepts discussed in the paper provide meaningful approaches to address the issue.\n\nOverall, the paper touches the important question of distributional shift for machine learning systems but I find the concepts discussed in the paper, such as the focus on meta-learning, the \"unit-test\", and \"context-swapping\", less relevant to how we can really mitigate the issues in real systems or how it can provide additional insights about the problem.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1e8RBdx9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main idea of the paper: When using meta-learning there is an inherent incentive for the learner to win by making the task easier. The authors generalise this effect to a larger class of problems where the learning framework induces a set of Hidden Incentive for Distributional Shift (HIDS) and introduce Context Swapping, a HIDS mitigation technique. In the experimental section, the authors propos a HIDS unit test which then they employ to show that PBT (Population Based-Trainng), a popular meta-learning algorithm exhibits HIDS ant that context swapping helps fixing it. \n\nOverall, I found the idea of the paper interesting, but the attempt to generalise the effect from meta-learning to general learning setups hard to follow and detracting from the overall value. I think the authors should have restricted their claims to PBT and PBT-like methods and follow-up with something more general in future work. \n\nFurthermore, the notation and formaliation of the problem are incomplete:\n    * the concept of ‘trajectory’ is introduced without being properly defined, though its crucial in the definition of the proposed HIDS mitigation approach\n    * the context swapping algorithm description is not clearly motivated and explained, a diagram showing the learner shuffling would be quite helpful\n    * in the HIDS unit-test section, the game theoretical setup is only partially explained, the defection and cooperation actions are not clearly linked to the HIDS \n    * In Section 4.1.1 HIDS UNIT TEST EXPERIMENTAL RESULTS AND DISCUSSION Figure2 is refered for results without ever stating the task and the Figure itself does not mention it\n\n       In terms of suggestions, I think the paper needs to go through a careful refactoring with an attention to technical details (careful concept definition, introduction of notation, clarity on experimental setup) ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows how meta-learning contains hidden incentives for distributional shift and how a technique called context swapping can help deal with this. Overall, distributional shift is an important problem, but the contributions made by this paper to deal with this, such as the introduction of unit-tests and context-swapping, is not sufficiently clear. Therefore, my recommendation is a reject.
Decision: Reject