user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Hidden incentives for self-induced distributional shift

Paper content:
Decisions made by machine learning systems have increasing influence on the world.
Yet it is common for machine learning algorithms to assume that no such influence exists.
An example is the use of the i.i.d.
assumption in online learning for applications such as content recommendation, where the (choice of) content displayed can change users' perceptions and preferences, or even drive them away, causing a shift in the distribution of users.
Generally speaking, it is possible for an algorithm to change the distribution of its own inputs.
We introduce the term self-induced distributional shift (SIDS) to describe this phenomenon.
A large body of work in reinforcement learning and causal machine learning aims to deal with distributional shift caused by deploying learning systems previously trained offline.
Our goal is similar, but distinct: we point out that changes to the learning algorithm, such as the introduction of meta-learning, can reveal hidden incentives for distributional shift (HIDS), and aim to diagnose and prevent problems associated with hidden incentives.
We design a simple  environment as a "unit test" for HIDS, as well as a content recommendation environment which allows us to disentangle different types of SIDS.
We demonstrate the potential for HIDS to cause unexpected or undesirable behavior in these environments, and propose and test a mitigation strategy.
Consider a household robot, one of whose duties is to predict when its owner will ask it for coffee.
We would like the robot to notice its owners preference for having coffee in the morning, but we would not want the robot to prevent its owner from sleeping late just because the robot is unsure if the owner will still want coffee if they wake up in the afternoon.
While doing so would result in a better prediction, such a strategy is cheating -by changing the task rather than solving the task as intended.
More specifically, waking the owner is an example of what we call self-induced distributional shift (SIDS), as it changes the distribution of inputs to the robot's coffee prediction algorithm.
SIDS is not necessarily undesirable: consider an algorithm meant to alert drivers of imminent collisions.
If it works well, such a system will help drivers avoid crashing, thus making self-refuting predictions which result in SIDS.
What separates this example from the coffee robot that disturbs its owner's sleep?
The collision-alert system alters its data distribution in a way that is aligned with the goal of fewer collisions, whereas the coffee robot's strategy results in changes that are misaligned with the goal of good coffee-timing (Leike et al., 2018) .
This makes it an example of a specification problem (Leike et al., 2017; Ortega & Maini, 2018 ): we did not intend the robot to ensure its predictions were good using such a strategy, yet a naive specification (e.g. maximizing likelihood) incentivized that strategy.
Ideally, we'd like to specify which kinds of SIDS are acceptable, i.e. the means by which a learner is intended or allowed to influence the world in order to achieve its' ends (i.e. increase its performance), but doing so in full generality can be difficult.
An alternative, more tractable problem which we address in this work is to accept the possibility of SIDS, but to carefully manage incentives for SIDS.
Informally, a learner has an incentive to behave in a certain way when doing so can increase its performance (e.g. higher accuracy, or increased reward).
When meta-learning optimizes over a longer time horizon, or using a different algorithm, than the original "inner loop" learner, this can reveal new incentives for SIDS that were not apparent in the original learner's behavior.
We call these hidden incentives for distributional shift (HIDS), and note that keeping HIDS hidden can be important for achieving aligned behavior.
Notably, even in the absence of an explicit meta-learning algorithm machine learning practitioners employ "manual meta-learning", also called "grad student descent" (Gencoglu et al., 2019) in the iterative process of algorithm design, model selection, hyperparameter tuning, etc.
Considered in this broader sense, meta-learning seems indispensable, making HIDS relevant for all machine learning practitioners.
A real-world setting where incentives for SIDS could be problematic is content recommendation: algorithmically selecting which media or products to display to the users of a service.
For example (see Figure 1 ), a profit-driven algorithm might engage in upselling: persuading users to purchase or click on items they originally had no interest in.
Recent media reports have described 'engagement'-(click or view-time) driven recommendation services such as YouTube contributing to viewer radicalization (Roose, 2019; Friedersorf, 2018) .
A recent study supports these claims, finding that many YouTube users "systematically migrate from commenting exclusively on milder content to commenting on more extreme content" (Ribeiro et al., 2019) .
1 See Appendix 1 for a review of real-world issues related to content recommendation.
Our goal in this work is to show both (1) that meta-learning can reveal HIDS, and (2) that this means applying meta-learning to a learning scenario not only changes the way in which solutions are searched for, but also which solutions are ultimately found.
Our contributions are as follows:
1. We identify and define the phenomena of SIDS (self-induced distributional shift) and HIDS (hidden incentives for distributional shift).
2. We create two simple environments for studying identifying and studying HIDS: a "unit test" based on the Prisoner's Dilemma, and a content recommendation environment which disentangles two types of SIDS.
3. We demonstrate experimentally that meta-learning reveals HIDS in these environments, yielding agents that achieve higher performance via SIDS, but may follow sub-optimal policies.
4. We propose and test a mitigation strategy based on swapping learners between environments in order to reduce incentives for SIDS.
We first show that agents trained with PBT fail the unit test more often when compared with baseline agents that do not use meta-learning.
We use REINFORCE (Williams, 1992) with discount factor γ = 0 as the IL optimizer for these experiments.
Policies are represented by a single realvalued parameter θ (initialized as θ ∼ N (0, 1)) passed through a sigmoid whose output represents P (a t = defect).
PBT (with default settings, see Section 2.2) is used to tune the learning rate, with reward on the final time-step of the interval as the performance measure for PBT.
We initialize the learning rate log-uniformly between 0.01 and 1.0 for all experiments (whether using PBT or not).
We expect and confirm that the following two factors lead to higher rates of failure (cooperation):
1. Shorter intervals: These give the OL more opportunities to influence the population.
2. Larger populations: These make outliers with exceptional non-myopic performance more likely, and OL makes them likely to survive and propagate.
The baseline (no PBT) agents pass the unit test: P (cooperate) (averaged over agents) is close to 0% -see blue curves in Figure 2 .
However, despite the disincentive for cooperation and the myopic inner loop, agents trained with PBT and large populations fail the unit test: P (cooperate) is around 90% -see the top right subplot of Figure 2 .
Furthermore, we verify that context swapping significantly mitigates the effect of HIDS, decreasing undesirable cooperate behaviour to near-baseline levels -see bottom rows of Figure 2 .
This effect can be explained as follows: Because context swapping transfers the benefits of a learner's action to the next learner to inhabit that environment, it increases that learner's fitness, and thereby reduces the relative fitness (as evaluated by PBT's EXPLOIT step) of the non-myopic cooperate behaviour.
We observe some interesting exceptions with the combination of small populations and short PBT intervals.
Although context swapping still significantly decreases the effect of HIDS, non-myopic cooperate behaviour is observed as much as 20% of the time (for #learners=10, T = 1; see bottom-left plot).
We also observe that PBT reveals HIDS even when T = 1. We provide a detailed explanation for how this might happen in Appendix 3.1.2.
But we also note that for T = 1, the explanation that PBT operates on a longer time horizon than the inner loop does not apply, making it especially surprising that HIDS are revealed.
Thus we hypothesize that there are at least 2 mechanisms by which PBT is revealing HIDS: (1) optimizing over a longer time-scale, and (2) picking up on the correlation between an agent's current policy and the underlying state.
Mechanism (2) can be explained informally as reasoning as: "If I'm cooperating, then I was probably cooperating on the last time-step as well, so my reward should be higher".
As support for these hypotheses, we run control experiments identifying two algorithms (each sharing only one of these properties) that can fail the unit test (although context swapping remains effective):
1. Optimizing over a longer time-scale: replacing PBT with REINFORCE as an outer-loop optimizer.The outer-loop optimizes the parameters to maximize the summed reward of the last T time-steps.
As with PBT, we observe non-myopic behavior, but now only when T > 1. This supports our hypothesis that the exploitation of HIDS is due not to PBT in particular, but just to the introduction of sufficiently powerful meta-learning.
See Figure 2 for results.
2. Exploiting correlation: Q-learning with γ = 0 an = 0.1-greedy behavior policy and no meta-learning.
If either state was equally likely, the Q-values would be the average of the values in each column in Table 1 , so the estimated Q(defect) would be larger.
But the -greedy policy correlates states and actions, so the top-left and bottom-right entries carry more weight in the estimates, sometimes causing Q(defect) ≈ Q(cooperate) and persistent nonmyopic behavior.
See Figure 3 for results, Appendix 3.1.4 for more results, and Appendix 3.1.3 for important experimental details.
Lower is better, since the goal is for non-myopic incentives to remain hidden.
Despite making the inner loop fully myopic (γ = 0), both outer-loop optimizers reveal HIDS, however, leading agents to choose the cooperate action (top rows of
(a) and
(b)).
Environment-swapping significantly mitigates HIDS (bottom rows of
(a) and
(b)).
Figure 3: Q-learning sometimes fails the unit test; empirical p(cooperate) stays around 80-90% in 3 of 5 experiments (bottom row).
Each column represents an independent experiment.
Q-values for the cooperate and defect actions stay tightly coupled in the failure cases (col. 1,2,5), while in the cases passing the unit test (col. 3,4) the Q-value of cooperate is driven down over time.
Our recommender system is a 1-layer MLP trained with SGD-momentum.
Actions are sampled from the MLP's predictive distribution.
For PBT, we use T = 10 and 20 agents, and use accuracy to evaluate performance.
We run 20 trials, and match random seeds for trials with and without PBT.
See Appendix 3.2.2 for full experimental details.
We find that PBT yields significant improvements in training time and accuracy, but also greater distributional shift; see Figure 4 .
User base and user interests both change faster with PBT, and in particular user interests change more overall.
We observe that the distributions over user types typically saturate (to a single user type) after a few hundred time-steps (Figure 1; Figure 4 , Right).
We run long enough to reach such states, to demonstrate that the increase in SIDS from PBT is not transitory.
The environment has a number of free parameters, and our results are qualitatively consistent so long as (1) the initial user distribution is approximately uniform, and (2) the covariate shift rate (α 1 ) is faster than the concept shift rate (α 2 ).
See Appendix 3.2.4 for details.
We measure concept shift (change in P (y|x)) as the cosine distance between each user types' initial and current interest vectors.
And we measure covariate shift (change in P (x)) as the KL-divergence between the current and initial user distributions, parametrized by g 1 and g t , respectively.
In Figure  5 , we plot concept shift and covariate shift as a function of accuracy.
We observe that for both types of SIDS, at low levels of accuracy PBT actually causes less shift than occur in baseline agents; HIDS are only observed for accuracies above 60%.
This suggests that only relatively strong performers are able to pick up on the HIDS revealed by PBT.
See
We have identified the phenomenon of self-induced distributional shift (SIDS), and the problems that can arise when there are hidden incentives for algorithms to induce distributional shift (HIDS).
Our work highlights the interdisciplinary nature of issues with real-world deployment of ML systems -we show how HIDS could play a role in important technosocial issues like filter bubbles and the propagation of fake news.
There are a number of potential implications for our work:
1. When HIDS are a concern, our methodology and environments can be used to help diagnose whether and to what extent the final performance/behavior of a learner is due to SIDS and/or incentives for SIDS, i.e. to quantify their influence on that learner.

assistant: text: The paper generalizes the inherent incentive for the learner to win by making the task easier in meta-learning to a larger class of problems.