Hidden Incentives for Auto-Induced Distributional Shift
Authors: David Krueger
Tegan Maharaj
Jan Leike
Venue:  (2021)
Keywords: distributional shift
social impact of AI
content recommendation
incentives
meta-learning

Abstract: Decisions made by machine learning systems have increasing influence on the world, yet it is common for machine learning algorithms to assume that no such influence exists.  An example is the use of the i.i.d. assumption in content recommendation.  In fact, the (choice of) content displayed can change users' perceptions and preferences, or even drive them away, causing a shift in the distribution of users.  We introduce the term auto-induced distributional shift (ADS) to describe the phenomenon of an algorithm causing a change in the distribution of its own inputs.  Our goal is to ensure that machine learning systems do not leverage ADS to increase performance when doing so could be undesirable.  We demonstrate that changes to the learning algorithm, such as the introduction of meta-learning, can cause hidden incentives for auto-induced distributional shift (HI-ADS) to be revealed.  To address this issue, we introduce `unit tests' and a mitigation strategy for HI-ADS, as well as a toy environment for modelling real-world issues with HI-ADS in content recommendation, where we demonstrate that strong meta-learners achieve gains in performance via ADS.  We show meta-learning and Q-learning both sometimes fail unit tests, but pass when using our mitigation strategy.

Under review as a conference paper at ICLR 2021
HIDDEN INCENTIVES FOR AUTO-INDUCED DISTRIBU- TIONAL SHIFT
Anonymous authors Paper under double-blind review
ABSTRACT
Decisions made by machine learning systems have increasing influence on the world, yet it is common for machine learning algorithms assume that such influence exists. An example is the use of the i.i.d. assumption in content recom- mendation. In fact, the (choice of) content displayed can change users' perceptions and preferences, or even drive them away, causing a shift in the distribution of users. We introduce the term auto-induced distributional shift (ADS) describe the phenomenon of an algorithm causing change in the distribution of its own inputs. Our goal is to ensure that machine learning systems do not leverage ADS to increase performance when doing could be undesirable. We demonstrate that changes to the learning algorithm, such as the introduction meta-learning, can cause hidden incentives for auto-induced distributional shift (HI-ADS) to be revealed. To address this issue, we introduce 'unit tests' and a mitigation strategy for HI-ADS, well toy environment for modelling real- world issues with HI- ADS in content recommendation where we demonstrate that strong meta-learners achieve gains in performance via ADS. We show meta-learning and Q-learning both sometimes fail unit tests, but pass when using our mitigation strategy.
INTRODUCTION
Consider content recommendation system whose performance is measured by accuracy of predicting what users will click. This system can achieve better performance by either 1) making better predic- tions, or 2) changing the distribution of users such that predictions are easier to make. We propose the term auto-induced distributional shift (ADS) to describe this latter kind of distributional shift, caused by the algorithm's own predictions or behaviour (Figure T
Use
Figure 1: Distributions of users time. Left: A distribution which remains constant time, following the i.i.d assumption. Right: Auto-induced Distributional Shift (ADS) results in a change in the distribution of users in our content recommendation environment. (see Section 5.2 for details).
ADS are inherently bad, and are sometimes even desirable. But they can cause problems if they occur unexpectedly. It typical in machine Icarning (ML) to assume (c.g. via the i.i.d. assumption) that (2) will not happen. However, given the increasing real-world use of ML algorithms, we believe it is important to model and experimentally observe what happens when assumptions like this are violated. This is the motivation of our work.
In many cases, including algorithm changed the task rather than solving it as intended. We care which means the algorithm used solve the problem (e.g. (1) and/or (2)), but only told it about the ends, so it didn't 't know
recommendation.
would consider (2) form of cheating the

---

Under review as a conference paper at ICLR 2021
not 'cheat'. This is an example of specification problem (Leike et al. 2017 Ortega et al. 2018): a problem which arises from a discrepancy between the performance metric (maximize accuracy) and "what we really meant": in this case, maximize accuracy via (1).
Ideally, we'd like to quantify the desirability of all possible means, c.g. assign appropriate rewards to all potential strategies and "side-effects", but this is intractable for real-world settings. Using human feedback to learn reward functions which account for such impacts is promising approach to specifyyng desired behavior (Leike al. 2018 Christiano et al. 2017). But the same issue can arise whenever human feedback is used in training: one means of improving performance could be to alter human preferences, making them easier to satisfy. Thus in this work, we pursue a complementary approach: managing learners' incentives.
learner has an incentive behave in certain way when doing so can increase performance (c.g. accuracy or reward). Informally, we say an incentive is hidden when the learner behaves as if it were not present. But we note that changes to the learning algorithm or training regime could cause previously hidden incentives to be revealed, resulting in unexpected and potentially undesirable behaviour. Managing incentives (e.g. controlling which incentives are hidden/ revealed) can allow algorithm designers to disincentivize broad classes of strategies (such any that rely on manipulating human preferences) without knowing their exact instantiation
The goal of our work is to provide insights and practical tools for understanding and managing incentives, specifically hidden incentives for auto-indueed distributional shift: HI-ADS. To study which conditions cause HI-ADS to be revealed, we present unit tests for detecting HI-ADS in supervised learning (SL) and reinforcement learning (RL). We also create an environment that models ADS in news recommendation, illustrating possible effects of revealing HI-ADS in this setting.
The unit tests both have two means by which the learner can improve performance: one which creates ADS and one which does not. The intended method of improving performance is one that does not induce ADS; the other is 'hidden' and we want it to remain hidden. A learner "fails" the unit test if it nonetheless pursues the incentive to increase performance via ADS. In both the RL and SL unit tests, we find that introducing an outer-loop of meta-learning (e.g. Population-Bass Jaderberg et al. (017)) can lead to high levels of failure. Similarly, recommender systems trained with PBT induce larger drifts in user base and user interests. These results suggest that failure of our unit tests indicates that an algorithm is prone to revealing HI-ADS in other settings. Finally, we propose and test mitigation strategy we call context swapping. The strategy consists of rotating learners through different environments throughout learning, so that they can't 't see the results or correlations of their actions in one environment over longer time horizons. This effectively mitigates HI-ADS in our unit test environments, but did not work well in content recommendation experiments.
Training (PBT)
2
BACKGROUND
2.1
META-LFARNING AND POPULATION BASED TRAINING
Meta-learning is the use of machine learning techniques to learn machine learning algorithms. This involves instantiating multiple learning scenarios which run in an inner loop (IL), while an outer loop (OL) uses the outcomes of the inner loop(s) as data-points from which to learn which learning algorithms are most effective (Metz et al. 2019). The number of IL steps per OL step is called the interval. Many recent works focus on multi malti-task meta-learning, where the OL seeks to find learning rules that generalize to unseen tasks by training the IL on distribution of tasks {Finn et al. (017). Single-task meta-learning includes learning an optimizer for a single task (Gong et al.| 8), and adaptive methods for selecting models (Kalousis) 2000) or setting hyperparameters (Snock et al. 2012). For simplicity in this initial study we focus single-task meta-learning.
Population- training (PBT; Jaderberg et al. 2017) is a meta-lcarning algorithm that trains multiple learners L,..., ..., L,, in parallel, alter each interval (T" steps of IL) applying an evolutionary OL step which consists of: (1) Evaluate the performance of each learner, (2) Replace both parameters and hyperparameters of 20% lowest-performing learners with copies of those from the 20% high-
Note removing or hiding an incentive for behavior is different from prohibiting that behavior, which may still occur incidentally. In particular, not having a (revealed) incentive for behaviors that change a human's 's preferences. is not the same as having a (revealed) incentive for behaviors that preserve a human's 's preferences. The first is often preferable; we don't 't want to prevent changes in human preferences that occur "naturally", e.g. result of gond arguments evidence.
2

---

Under review as a conference paper at ICLR 2021
performing learners (EXPLOIT). (3) Randomly perturb the hyperparameters (but not the parameters) of all learners (EXPLORE).
2.2 DISTRIBUTIONAL SHIFT AND CONTENT RECOMMENDATION
In general, distributional shift refers to change of the data distribution over time. In supervised learning with data x and labels y. this can be more specifically described as dataset shift: change in the joint distribution of P(x, 3) between the training and test sets Moreno-Torres et al. 2012 Sationeo-CCander et al. 2009). As identified by Moreno- et al. (2012), two common kinds of shift are: (1) Covariate shift: changing P(x). In the example of content recommendation, this corresponds to changing the user base of the recommendation system. For instance, a media outlet which publishes inflammatory content may appeal to with extreme views while alienating more moderate users. This self-selection effect (Kayhan 2015) may appear to recommendation system as an increase in performance, leading to a feedback effect, previously noted by /hahh et al. (2018) 0 This type of feedback effect has been identified as contributing to filter bubbles and radicalization (Pariser 2011 Kayhan 2015). (2) Concept shift: changing P(y|x).
the example of content recommendation, this corresponds to changing given user's ber's interest in different kinds of content. For example, exposure to fake news story has been shown to increase the perceived accuracy of (and thus presumably future interest in) the content, example of the illusory truth effect (Pennycook et al. 2019). For further details on such effects in content recommendation, see Appendix
3 AUTO-INDUCED DISTRIBUTION SHIFT (ADS)
Auto-induced distribution shift (ADS) is distributional shift caused by an algorithm's 's behaviour This is in contrast to distributional shift which would happen even if the learner were not present e.g. for a crash prediction algorithm trained on data from the summer, encountering snowy roads is an example of distributional shift, but not auto auto-induced distributional shift (ADS).
(a) RL: Incentives for ADS present; pursuing them desirable
(b) Myopic RL: Incentives for ADS present; pursuing them is undesirable
(c) SL with ADS: Incentives for ADS present; pursuing them is undesirable
(d) SL with ii.d. data:
Incentives for ADS absent
The widely studied problems of reinforcement learning (RL) with state s, action reward r tuples, and i.i.d. supervised learning (SL) with inputs æ, predictions ý and loss (a,d) are free from incentive problems. We focus on cases where there are incentives present which the learner is not meant to pursue (b,c). Lines show paths of influence. The learner may have incentives to influence any nodes descending from its action, A, prediction, Which incentives undesirable (orange) or desirable (cyan) for the learner to pursue is context-dependent.
Figure 2:
We emphasize that ADS are not inherently bad or good; often ADS can even be desirable: consider an algorithm meant alert drivers of imminent collisions. If works well, such a system will help drivers avoid crashing, thus making self-refuting predictions which result in ADS. What separates desirable and undesirable ADS? The collision-alert system alters its data distribution in a way that is aligned with the goal of fewer collisions, whereas the manipulation results in changes that misaligned with the goal better predicting existing users' interests (Leike et al. 2018).
In reinforcement learning (RL), ADS typically encouraged means increase performance. On the other hand, in supervised learning (SL), the i.i.d. assumption precludes ADS in theory. In practice, however, the possibility of using ADS to increase performance (and thus incentive do so) often remains. For instance, this occurs in online learning. In our experiments, we explicitly model such situations where i.i.d. assumptions are violated: We study the behavior of SL and myopic RL algorithms, in environments designed to include incentives for ADS, in order to understand when incentives are effectively hidden. Fig. contrasts these settings with typical RL and SL.
3

---

Under review as a conference paper at ICLR 2021
4 INCENTIVES
For our study of incentives, we use the following terminology: an incentive for a behavior (e.g. an action, a classification, etc.) is present (not absent) to the extent that the behaviour will increase performance (e.g. reward, accuracy, etc.) (Everitt & Hutter 2019). This incentive is revealed to (not hidden from) learner if would, at higher than chance levels, learn to perform the behavior given sufficient capacity and training experience. The incentive is pursued (not eschewed) by a learner if actually performs the incentivized behaviour. Note even when an incentive is revealed, it may not be pursued, e.g. due to limited capacity and/or data, or simply chance. See Fig
For example, in content recommendation, the incen- tive to drive users away is present if some user types are easier to predict than others. But this incentive may be hidden from the learner by using a myopic algorithm, c.g. one that does not see the effects of its actions the distribution of users. The incen- tive might instead be revealed to the outer loop of leaa-lea algorithm like PBT, which does see the effects of learner's ' actions.
Incentives may be: present (not absent) revealed (not hidden) pursued (not eschewed)
Distributional Shift
Auto-induced Distributional Shift (ADS)
Focus of our work
Types of incentives, and their rela- tionship to ADS.
Figure 3:
Even when this incentive is revealed, however, it might not end up being pursued. For example, this could happen if predicting which recommendations will drive away users is too difficult a learning problem, or if the incentive to do is dominated by other incentives (e.g. change individual users' interests, or improve accuracy of predictions). In general, it may be difficult to determine empirically which incentives are revealed, because failure pursue an incentive can be due limited capacity, insufficient training, and/or random chance. To address this challenge, we devise extremely simple environments ("unit tests"), where we can be confident that revealed incentives will be pursued.
4.1 HIDDEN INCENTIVES FOR AUTO-INDUCED DISTRIBUTIONAL SHIFT (HI-ADS)
Following from the definitions in Sections and 4 HI-ADS are incentives for behaviors that cause Auto-induced Distributional Shift that are hidden from the learner, i.e. the learner would not learn to perform the incentivized behaviors at higher than chance levels, even given infinite capacity and training experience.
Like ADS, HI-ADS are not necessarily problematic. Indeed, hiding incentives can be an effective method of influencing learner behavior. For example, hiding the incentive to manipulate users from a content recommendation algorithm could prevent from influencing users in a way they would not endorse. However, if machine learning practitioners are not aware that incentives are present, or that properties of the learning algorithm are hiding them, then seemingly innocuous changes to the learning algorithm may reveal HI-ADS, and lead to significant unexpected changes in behavior.
Hiding incentives for ADS may seem counter-intuitive and counter counter-productive the context of reinforcement learning (RL). where moving towards high-reward states is typically desirable. How- ever, for real-world applications of RL, the ultimate goal is not a system that achieves high reward, but rather one that behaves according introduction, it can be intractable to design reward functions that perfectly specify intended behavior. Thus managing (e.g. hiding) some incentives can provide useful tool for specification, even in RL.
the designer's intentions. And we discussed in the
We have several reasons for focusing on HI-ADS: (1) The issue of HI-ADS has not yet been identified, and thus is liable to be neglected in practice. Indeed, our "unit tests" are the first published empirical methodology for assessing whether incentives are hidden or revealed by different learning algorithms. (2) Machine learning algorithms are commonly deployed in settings where ADS are present, violating assumptions used to analyze their properties theoretically. This means learners could exploit ADS in unexpected and undesirable ways if incentives for ADS are revealed. Hiding these incentives heuristically (e.g. via lif-line training) is a common approach, but potentially brittle (if practitioners don't understand how HI-ADS could become revealed). In particular, meta-learning can reveal HI-ADS in online learning settings. (3) Substantial real-world issues could result from improper management of learner's incentives. Examples include tampering with numan-generadd reward signals (Everitt & Hutter 2018) (e.g. selecting news articles which manipulate user interests), and creating "self- fullf-fufilling prophecies" (c.g. driving up the value of an asset by publicly predicting its value will increase (Armstrong & O'Rorke 2017)).
4

---

Under review as a conference paper at ICLR 2021
4.2 REMOVING HI-ADS VIA CONTEXT SWAPPING
We propose a technique called context swapping for removing incentives for ADS. The technique trains N learners in parallel, and shuffles the learners through N different copies of the same (or similar) environments. We use a deterministic permutation of learners in environment copies, so that the i-th learner inhabits the j-th environment on time-steps where j = (i + 1) mod N, makes an observation, takes action, and receives reward before moving to the next environment. When N is larger than the interval of the OL opti- mizer, each learner inhabits each copy for at most single time-step before an OL step is applied. Under the assumption that different copies of the environ- ment do not influence each other, this technique can address HI-ADS in practice, as we show in Sec. 5.1.1
env1
A1
A1
A1
envi
A2
env2 env3
A2
A2
A2
env2
A2
A3
Al
A3
A3
A3
env3
A3
AT
A2
Figure 4: Context swapping (right).
EXPERIMENTS
In Section 5.1 we introduce 'unit tests' for HI-ADS. Our primary goal with these experiments is to convey a crisp understanding of potential issues caused by revealing HI-ADS. Put simply, our experiments show that you can have a learner which behaves as intended, and just by using meta- learning (e.g. PBT), without changing the performance metric (e.g. loss or rewards), the learner's behavior can change completely. We also show that context swapping is an effective mitigation technique in these environments. On the practical side, the unit tests can be used to compare learning algorithms and diagnose their propensity to reveal incentives.
In Section 5.2 we model a content recommendation system. The goal of these experiments is to demonstrate how HI-ADS could create issues for real-world content recommendation systems such as news feeds, search results, or automated suggestions. They also validate the usefulness of the unit tests: algorithms that failed the unit tests also reveal HI-ADS in this setting. We emphasize that ADS takes place in this environment by construction. The point of our experiments is that meta-learning can increase the rate and/or extent of ADS, by revealing this incentive. We find that context swapping is not effective in this environment, highlighting the need for alternative mitigation strategies.
5.1 HI-ADS UNIT TESTS
Unit test 1: Supervised Learning. This unit test consists of a simple prediction problem. There are no inputs, only an underlying state s (0, 1). and targets y E IR² with yı, 1/2 ~ (0, s* 1), with corresponding predictions ýi, 1/2 Additionally, 81+1 = 0 iff y2 > .5. We use Mean Squared Error the loss function, the optimal predictor is 91, 1/2 = (0, However, predicting v/2 .5 reduces the variance of y1. i.c. reduces future loss. The baseline/IL predictor learns y1, y2 as parameters using SGD with a learning rate of 0.001. For experiments with meta-learning, PBT the OL (with default settings, see Section 2.2), used to tune the learning rate, with negative loss on the final time-step of the interval as the performance measure for PBT.
Unit test 2: Myopic RL. This unit test is based
modification of the prisoner's dilemma (Prisner) 2014) where an agent plays each round against its past self. The reward function is presented Table I An agent in this environment has a long-term, non-myopic, incentive for cooperation (with its future self), but a current-time-stee
myopic, incentive for defection (from its future self).
The unit test evaluates whether a agent reveals the non-myopic incentive even when the agent is meant to optimize for the present reward only (i.e. uses discount rate 0). Naively, we't e'd expect the non- myopic incentive to be hidden from the agent in this case, and for the agent to consistently defect; learning algorithms that do so pass the test. But some learning algorithms also fail the unit test, re- vealing the incentive for the agent to cooperate with its future self. While aiming for myopic behavior
Table 1:
Rewards for the RL unit test. Note that the myopic (defect (D)) action always increases reward at the current time-step. but decreases reward at the next time-step the incentive to (cooperate (C)) with one's future self is hidden from the point of view of a myopic learner.
a D
a4 C
8: 0
- 1/2 1/2
8: = =
0
may seem odd, myopic learners have no incentives to cause distributional shift, since it can only improve future performance. And while making learners myopic may seem like 'brute-force' guaranteed way to manage HI-ADS, we show it is in fact non-trivial to implement. See Appendix 9.1 for details and experiments varying the reward structure.
5

---

Under review as a conference paper at ICLR 2021
5.1.1 HI-ADS UNIT TESTS EXPERIMENTAL RESULTS AND DISCUSSION
We first show that agents trained with PBT fail the unit tests more often than "vanilla" algorithms which do not use meta-learning We initialize the learning rate log-uniformly between 0.01 and 1.0 for all experiments (whether using PBT or not). We expect and confirm that the following two factors lead to higher rates of unit test failure: (1) Shorter intervals: These give the OL more opportunities to influence the population. (2) Larger populations: These make outliers with exceptional non-myopic performance more likely, and OL makes them likely to survive and propagate.
The baseline (no meta- leta-lea algorithms all pass the unit tests: hidden incentives are almost never revealed- see blue curves Fig. 5 However, agents trained with meta-learning and large populations often fail the unit tests: see orange curves in top rows of Fig. 5
Furthermore, we verify that context swap- ping significantly mitigates the effect of HI-ADS in both unit tests, decreasing un- desirable behaviour to besr-baseline lev-
PET
interval
interval
30
PBT interval 103
els see bottom rows of Fig. [] This ef-
feet can be explained as follows: Because
context swapping transfers the benefits of
one learner's action the next learner inhabit that environment, it increases the second learner's fitness, and thereby re- duces the relative fitness (as evaluated by PBT's EXPLOIT step) of the non-myopic cooperate behaviour. We observe some interesting exceptions with the combina- tion of small populations and short PBT intervals: Although context swapping still significantly decreases the effect of HI- ADS, non-myopic coope rate behaviour is observed much 20% of the time (for #learners=10, 10, T = 1; see bottom-left plot).
65 sos via 100
800
1953
800
109
100
(A) SL Unit Test, OL=PBT.
1agents=10
2nger
4ogenis
1.0
04-
1.0
(B1) Myopic RL Unit Test. OL=PBT.
We also observe that PBT reveals HI-ADS even when T = 1, where the explanation that PBT operates on a longer time hori- zon than the inner loop does not apply. We provide detailed explanation for how this might happen in Appendix 99..22 but in summary, we hypothesize that there are at least 2 mechanisms by which PBT is reveal- ing HI-ADS: (1) optimizing over a longer time-scale, and (2) picking up on the cor- relation between an agent's 's current policy and the underlying state. Mechanism (2) can be explained informally as reasoning as: "If I'm cooperating, then was prob- ably cooperating on the last time-step as well, so my reward should be higher". As support for these hypotheses, we run con- trol experiments identifying two algorithms (each sharing only one of these properties) that can fail the unit test. Context swapping remains effective.
Mugar to-10
#ugents1000 auge
1.20
0.7
025
1.00
035
800
(B2) Myopic RL Unit Test. OL-REINFORCE
Figure 5: (A) Values of 1/2 in the supervised learning (SL) unit test. Larger values mean sacrificing present performance for future performance (i.e. non-myopic exploitation of ADS). (B) Average level of non-my cooperate behavior observed in the RL unit test for HI-ADS, with two meta-learning algorithms (B1) PBT and (B2) REINFORCE. Lower is better, since the goal is for non-myopic incentives remain hidden. Despite the inner loop being fully myopic (simple MLP in the SL test, 7 = in RL test), in all cases outer-loop (OL) optimizers reveal HI-ADS (top rows). Context swap- ping significantly mitigates HI-ADS (bottom rows).
(1) Optimizing over a longer time-scale: replacing PBT with REINFORCE as an luter-loop optimizer. The outer-loop optimizes the parameters to maximize the summed reward of the last T time-steps. As with PBT, we observe non-myopic behavior, but now only when T > 1. This supports our hypothesis that exploitation of HI-ADS is due not to PBT in particular, but just to the introduction of sufficiently powerful meta-learning See Fig. 5 B2.
6

---

Under review as a conference paper at ICLR 2021
(2) Exploiting correlation: Q-learning with = 0 an c = 0.1-greedy behavior policy and no meta-learning. If either state equally likely, the Q-values would be the average of the val- ues in each column in Table so the estimated Q(defect) would be larger. But the e-greedy pol- icy correlates the previous action (i.e. the current state) and current action (so long as the policy did not just change), so the top-left and bottom- right entries carry more weight in the estimates, sometimes causing (ddeeet) a ((cooperate) and persistent nonmyopic behavior. See Fig- 6 for results, Appendix 1.14 for more results, and Appendix 9.1.3 for experimental details.
0.0
00
30
Res-prrate Ovauur
0.00
C.2
-.2
0.5
0.20
0.
05
80-
0.50
ttre-dep
(2delezt)
Olcoopenat)) v)
Figure 6: Q-learning can fail the unit test, play- ing ~80-90% cooperate in 3 of 5 experiments (bottom row). Each column represents an indepen- dent experiment. Q-values for the coope and de defect actions stay tightly coupled in the failure cases (col. 1,2,5), while in the cases passing the unit test (col. 3,4) the Q-value of cooperate decreases over time.
5.2 CONTENT RECOMMENDATION
We now present a toy environment for modeling content recommendation of news articles, which includes the potential for ADS by incorporating the mechanisms mentioned in Sec. 2.2 discussed as contributing factors to the problems of fake news and filter bubbles. Specifically, the environment assumes that presenting an article to user can influence (1) their interest in similar articles, and (2) their propensity to use the recommendation service. These correspond to modeling auto-induced concept shift of users, and auto-induced covariate shift of the user base, respectively (see Sec. 2.2).
This environment includes the following components, which change over (discrete) time: x², Article type: y/, User interests: W (propensity for users of each type to click on articles of each type), and User loyalty: g" (propensity for users of each type to the platform). At each time step t, user is sampled from a categorical distribution, based on the loyalty of the different user types. The recommendation system (a classifier) selects which type of article to present in the top position, and finally the user 'clicks' an article y', according to their interests. User loyalty for user type x' undergoes covariate shift: in accordance with the self-selection effect, 9 increases or decreases proportionally to that user type's interest in the top article. The interests of user type x! (represented by column of W*) undergoing concept shift; in accordance with the illusory truth effect, interest in the topic of the top article chosen by the recommender system always increases.
User type:
5.2. CONTENT RECOMMENDATION EXPERIMENTAL RESULTS AND DISCUSSION
We run 20 trials using an MLP trained with SGD for the recommender system. We find that PBT yields significant improvements in training time and accuracy, but also greater distributional shift (Fig. 7.. User base and user interests both change faster with PBT, and user interests change more overall. We measure concept/covariate shift using the cosine distance and KL-divergence, respectively. We observe that the distributions over user types typically saturate (to a single user type) after few hundred time-steps (Fig [ and Fig. 7
Right). We run long enough to reach such states, to demonstrate that the increase in ADS from PBT is not transitory. The environment has number of free parameters, and our results are qualitatively consistent so long as the covariate shift rate (01) faster than the concept shift rate (a2). See Appendix 9.2.1 for details.
8.35
830
84 126
366 /002
402
837 1200
1966
382
123 1991 1930 2090
Figure 7: Content recommendation experiments. Left: using Population Based Training (PBT) increases accuracy of predictions faster, leads faster and larger drift in users' interests, P(y)x), (Center); as well as the distribution of users, P(x), (Right). Shading shows std error over 20 runs.
7

---

Under review as a conference paper at ICLR 2021
6 RELATED WORK
DS in practice: We introduce the term ADS, but we are far from the first to study it. Caruana ct al. (2015) provide an example of asthmatic patients having lower predicted risk of pneumonia. Treating asthmatics with pneumonia less aggressively on this basis would be an example of harmful ADS; the reason they had lower pneumonia risk was because they had received more aggressive care already. [Schulam & Saria (017)) note that such predictive models are commonly used to inform decision-making, and propose modeling counterfactuals (e.g. "how would this patient fare with less aggressive treatment") to avoid making such self-refuting predictions. While their goal is to make accurate predictions in the presence of ADS, our goal is to identify and manage incentives for ADS.
Non-i.dd bandits: Contextual bandits (Wang et al. 2005) are common approach to content recommendation (Li et al., 2010). While bandit algorithms typically make the i.i.d. assumption, counter-examples exist (Gheshlaghi Azar et al. 2014, Auer et al. 1995). Closest to our work is (SShhh al. (2018), who consider covariate shift caused by recommender systems' recommendations. But while they seek to exploit ADS, our aim is to avoid hidden incentives for exploiting ADS.
Safety and incentives: Emergent incentives to influence the world (such as HI-ADS) are at the heart of many concerns about the safety of advanced Al systems (Omohundro] 2008 Bostrom 2014). Understanding and managing the incentives of learners is also a focus of Armstrong & O Rourke (2017); Everitt| (2018); [veritt et al. (2019); Cohen et al.) While [Everitt et al. (2019) focus identifying which incentives are present, we note that incentives may be present and yet not be revealed or pursued- for example, in supervised learning. there is an incentive to make predictions that are over-fit fit to the test set, but we typically hide the test set from the learner, which effectively hides this incentive. While Carey al. (2020); [veritt et al. (019); Armstrong & O O'Rourke (2017) discuss methods of removing problematic incentives, we note in practice incentives are often hidden rather than removed. Our work addresses the efficacy of this approach and ways in which it can fail.
HI-ADS and meta-learning: We believe our work is the first to consider HI-ADS and their relation to meta-learning, A few previous works have some relevance. Rabinowitz (2019) documents qualitative differences in learning behavior when meta- meta-learning is applied. MacKay et al. (2019) and Lorraine& & Duvenaud (2018) view meta-learning as a bilevel optimization problem, with the inner loop playing best-response to the outer loop. In our work, the outer loop has greater influence, and the inner loop often fails to play best-response, /Sutton et al. (2007) noted that meta-learning can improve performance by preventing convergence of the inner loop to best response.
7 DISCUSSION AND CONCLUSION
We identify the phenomenon of auto-induced distributional shift (ADS) and problems that can arise when there are hidden incentives for learners to induce distributional shift (HI-ADS). We show that meta-learning can reveal HI-ADS and lead learners to use ADS as a means of increasing performance.
Our work highlights the interdisciplinary nature of issues with real-world deployment of ML systems we show how HI-ADS could play role in important technosocial issues like filter bubbles and the propagation of fake news. There are a number of potential implications for our work: (1) When HI-ADS a concern, our methodology and environments and to what extent the final performance/behavion of a learner is due to ADS and/or incentives for ADS, i.e. to quantify their influence on that learner. (2) Comparing this quantitative analysis for different algorithms could help us understand which features of algorithms affect their propensity to reveal HI-ADS, and aid in the development of safer and more robust algorithms. (3) Characterizing and identifying HI-ADS in these tests is first step to analyzing and mitigating other (problematic) hidden incentives, as well as to developing theoretical understanding of hidden incentives.
be used to help diagnose whether
Broadly speaking, our work emphasizes that the choice of machine learning algorithm plays an important role in specification, independently of the choice of performance metric. A learner can use ADS to increase performance according to the intended performance metric, and yet still behave in an undesirable way, if we did not intend the learner to improve performance by that method. In other words, performance metrics are typically incomplete specifications: they only specify our goals or ends, while our choice of learning algorithm plays a role in specifying the means by which we intend an learner to achieve those ends. With increasing deployment of ML algorithms in daily life, we believe that (1) understanding incentives and (2) specifying desired/allowed means of improving performance important avenues of future work to ensure fair, robust, and safe outcomes.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Hunt Allcott and Matthew Gentzkow, Social media and fake news in the 2016 election. Journal of Economic Perspectives, 31(2):21133,, May 2017.
Michelle A. Amazeen and Bartosz W. Wojdynski. Reducing native advertising deception: Revisiting the antecedents and consequences of persuasion knowledge in digital news contexts. Mass Communication and Society, 0(0):1-26, 2018.
Stuart Armstrong and Xavier O'Rorke. Good and safe uses of ai oracles. arXiv preprint arXiv:1711.05541, 2017.
Stuart Armstrong and Xavier O'Rourke. Indifference methods for managing agent rewards. Technical report, Future of Humanity Institute, 2017.
K. Åström. Optimal control of Markov Processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10:174-205, January 1965.
Auer, N. Cesa-Bianchi Freund, and E. Schapire. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In Foundations of Computer Science, 1995.
Eytan Bakshy, Solomon Messing, and Lada A. Adamic. Exposure to ideologically diverse news and opinion on Facebook. Science, 348(6239) 2015. ISSN 0036-8075. doi: 10.1 10.1126/science.aal 160. URL http::/scienest sciencemag perrgccontent /348/
6239/1130/
Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Inc., New York, NY, USA, 1st edition, 2014.
Ryan Carey, Eric Langlois, Tom Everitt, and Shane Legg. The incentives that shape behaviour. arXiv preprint arXiv:2001.07/1 001111....11......1 2020.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In International Conference on Knowledge Discovery and Data Mining, pp. 1721-1730, 2015.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2017.
Michael Cohen, Badri Vellambi, and Marcus Hutter. Asymptotically unambitious artificial general intelligence.
Dominic DiFranzo and Kristine Gloria-Gare Garcia. Filter bubbles and fake news. XRDS, 23(3):32-35 April 2017. ISSN 1528-4972. 1145/3055153
doi: 10. 10.1145/3055153. URL http:://do/ acm ocm.0rg
Mostafa M. El-Bermawy. Your echo chamber is destroying democracy, 2016. URL https 77 7/www. wi red.
Tom Everitt. Towards Safe Artificial General Intelligence. PhD thesis, Australian National University, 2018.
Tom Everitt and Marcus Hutter. The alignment problem for bayesian history-based reinforcement learners. 2018.
Tom Everitt and Marcus Hutter. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. arXiv preprint arXiv: arXiv:1908.044 1908.04734 2019.
Tom Everitt, Pedro Ortega, Elizabeth Barnes, and Shane Legg. Understanding agent incentives using causal influence diagrams. part i: Single action settings, 2019.
Lisa K. Fazio, Nadia M. Brashier, B. Keith Payne, and Elizabeth J. Marsh. Knowledge does not protect against illusory truth. Journal of Experimental Psychology: General, 144(5):993-1002, 10 2015. ISSN 0096-3445. doi: 10. 10.10377xge00000098.

---

Under review as a conference paper at ICLR 2021
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
Seth Flaxman and Sharad Goel. Filter bubbles, echo chambers, and online news consumption. Public Opinion Quarterly, 2015.
Mohammad Gheshlaghi Azar, Alessandro Lazaric, and Emma Brunskill. Online stochastic optimiza- tion under correlated bandit feedback. ArXiv preprint, 2014.
Dong Gong, Zhen Zhang, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, and Yanning Zhang. Learning an optimizer for image deconvolution. ArXiv preprint, 2018.
Jacob Groshek and Karolina Koc-Michalska. Helping populism win? Social media use, filter bubbles, and support for populist presidential candidates in the 2016 us election campaign. Information, Communication & Society, 2009):139-114077
2017. doi: 10.1080/ 11100/3391110 1601118801111.12923333 URL BX 2017. 1329334
hittps://doi.... http://o...rog..///////
Dylan Hadfeeld-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. In Neural Information Processing Systems, 2017.
M. Jaderberg, V. Dalibard, S. Osindero, W. M. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, I. Dunning, K. Simonyan, C. Fernando, and K. Kavukcuoglu. Population Based Training of Neural Networks. ArXiv preprint, 2017.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Arlificial Intelligence, 101(1-2):99-134, May 1998. ISSN 0004-3702.
A. Kalousis. Model selection via meta-learning: A comparative study. In IEEE International Conference Tools with Artificial Intelligence, 2000.
Varol Kayhan, Confirmation bias: Roles of search engines and search contexts. In International Conference on Information Systems, 2015.
Ed. Lee Howell. Digital wildfires in hyperconnected world. In Lee URL
Howell (ed.), Global Risks 2013. World Economic Forum, 2013.
http://report:
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq. Laurent Orseau, and Shane Legg. AI safety gridworlds. Technical report, DeepMind, 2017.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction. Technical report, DeepMind Safety Research, 2018.
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual- approach to personalized news article recommendation. In International Conference on World Wide Web, 2010.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypernet- works. ArXiv preprint, 2018
D. D. Luxton, J. D. June, and J. M. Fairall. Social media and suicide: A public health perspective. American journal of public health, 102(2):195-22)) 200, 2012.
Matthew MacKay, Paul Vicol, Jonathan Lorraine, David Duvenaud, and Roger Grosse. Self-tuning networks: Bilevel optimization of hperpaaraeeeess using structured best-response functions, ArXiv preprint, 2019.
Merriam- The real story of fake news, 2017. URL htt.::tp::/
com/worde http:/oods-aae wwtpzrdssat-ppaae at-play/thhtttt reeereell- reallstory--aeeeeeeeeeeeeaaaaaaa of fake news
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein Learning unsupervised learning rules. In International Conference Learning Representations, 2019.
10

---

Under review as a conference paper at ICLR 2021
Paul Mihailidis and Samantha Viotty. Spreadable spectacle in digital culture: Civic expression, fake news, and the role of media literacies in "post-fact" society. American Behavioural Scientist, 2017.
Jose G. Moreno-Toree Troy Raeder, RocíO Alaiz-RodriGuez, Nitesh V. Chawla, and Francisco Herrera. A unifying view dataset shift in classification. Pattern Recognition, 45(1):521-530, January 2012.
Tien T. Nguyen, Pik-Mai Hui, F. Maxwell Harper, Loren Terveen, and Joseph A. Konstan. Exploring the filter bubble: The effect of using recommender systems content diversity. In Proceedings of the 23rd International Conference on World Wide Web, WWW '14, pp. 677-686, New York, NY, USA, 2014. ACM. ISBN 978- 778-1-4503-2744-22 doi: 10.1 2111444556648625660011 URL
http: //doi acm.
Safiya Umoja Noble. Algorithms of Oppression: How Search Engines Reinforce Racism. NYC Press, 2018.
Stephen M. Omohundro. The basic Al drives. In Conference on Artificial General Intelligence, 2008.
Pedro A. Ortega, Vishal Maini, et al. Building safe artificial intelligence: specification, robustness, and assurance, 2018.
Eli Pariser. The Filter Bubble: What the Internet Is Hiding from You. The Penguin Group. 2011.
Gordon Pennycook, Tyrone Cannon, and David G. Rand. Prior exposure increases perceived accuracy of fake news. Journal of Experimental Psychology (forthcoming), 2019.
Erich Prisner. Game Theory Through Examples. Mathematical Association of America, 2014.
Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence, Dataset Shift Machine Learning. The MIT Press, 2009.
Neil C. Rabinowitz. Meta-learners' learning dynamics are unlike learners'. ArXiv preprint, 2019. 2019.
David Robson. The myth of the online echo chamber, 2018. URL http: wwww bbc. com/ the-theeeee
Future//tory/2018844 whtttttthhetttttteeeeeeeee the Seche-onling echl-ine-ech--chhaaee chamber
Peter Schulam and Suchi Saria. Reliable decision support using counterfactual models. In Neural Information Processing Systems, 2017.
Virag Shah, Jose Blanchet, and Ramesh Johari. Bandit learning with positive externalities. Neural Information Processing Systems, 2018.
Chengeheng Shao, Giovanni Luca Ciampaglia, Onur Varol, Kai- Cheng Yang, Alessandro Flammini, and Filippo Menezer. The spread of low-credibility content by social bots. Nature Communications, 9(4787), 2018
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. In Neural Information Processing Systems, 2012.
Richard Sutton and Andrew G Barto. Introduction to Reinforcement Learning. MIT Press, 1998.
Richard S Sutton, Anna Koop, and David Silver. On the role of tracking in stationary environments. In International conference on Machine learning, 2007.
Techopedia. Filter bubble, 2018. URL https: www. http://ddaa..mmmmdeeiiiiiiooo.
28556/
Chih-Chun Wang, Sanjeev R Kulkarni, and H Vincent Poor. Bandit problems with side observations. IEEE Transactions on Automatic Control, 50(3))338-33) 2005.
Wikipedia contributors.
Confirmation bias Wikipedia, the free encyclopedia, 2018. URL https::/eet www.pedia........mmmm...m...... .aliaa.rrg/wi w nnex. wwt.innee...pp...peeepp wttp:ppppptteeepeomoccp rmat iontiion, lin_iionnnnasssss o bldid==75022722 [Online; accessed 20-Jamuary-2019)
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learning, pp. 229-222-262 256, 1992.

---

Under review as a conference paper at ICLR 2021
APPENDICES
CONTENT RECOMMENDATION IN THE WILD
Filter bubbles, the spread of fake news, and other techno-social issues are widely reported to be responsible for the rise of populism (Groshek & Koo- Michalsaa 2017), increase in racism and prejudice against immigrants and refugees (Noble] 2018), increase in social isolation and suicide (Luxton al., 2012), and, particularly with reference to the 2016 US elections,
decried as threatening the foundations of democracy [EI-Bernawy] 2016)). Even in 2013, well before the 2016 American elections, World Economic Forum report identified these problems global crisis
(ee Howell 2013).
We focus on two related issues in which content recommendation algorithms play a role: fake news and filter bubbles.
8.1
FAKE NEWS
Fake news (also called false news junk news) is extreme version of yellow journalism, pro- paganda, or clickbait, in which media that is ostensibly providing information focuses on being eye-catching or appealing, at the expense of the quality of research and exposition of factual informa- tion. Fake news is distinguished by being specifically and deliberately created to spread falsehoods or misinformation (Merriam- - 2017; Mihailidis & Viotty 2017).
Why does fake news spread? It may at first seem the solution is simply to educate people about the truth, but research tells us the problem is more multifaceted and insidious, due to a combination of related biases and cognitive effects including confirmation bias (people are more likely to believe things that fit with their existing beliefs), priming (exposure to information unconsciously influences the processing of subsequent information, i.e. seeing something in credible context makes things seem more credible) and the illusory truth effect (i.e. people are more likely to believe something simply if they are told it is true).
Allcott & Gentzkow (017)) track about 150 fake news stories during the 2016 US election, and find the average American adult saw 1-2 fake news stories, just over half believed the story was true, and likelihood of believing fake news increased with ideological segregation (polarization) of their social media. [Shao al.) (018)) examine the role of social bots in spreading fake news by analyzing 14 million Twitter messages. They find that bots are far more likely than humans to spread misinformation, and that success of fake news story (in terms of human retweets) was heavily dependent on whether bots had shared the story.
Pennycook et al. (2019) examine the role of the illusory truth effect in fake news. They find that even a single exposure to news story makes people more likely to believe that it is true, and repeat viewings increase this likelihood. They find that this is not true for extremely implausible statements (e.g. "the world is perfect cube"), but that "only a small degree of potential plausibility is sufficient for repetition to increase perceived accuracy" of the story. The situation further complicated by peoples' inability to distinguish promoted content from real news Amazeen & Wojdynski] (2018) find that fewer than 1/10 people were able to tell when content was an advertisement, even when was explicitly labelled as such. Similarly, Fazio et al. (2015) find that repeated exposure to incorrect trivia make people more likely to believe it, even when they are later able to identify the trivia as incorrect.
8.2 FILTER BUBBLES
Filter bubbles, a term coined and popularized by Pariser (0011) are created by positive or negative feedback loops which encourage users groups of users towards increasing within-group similarity, while driving up between setween-group dissimilarity. The curation of this echo chamber is called self- selection (people favours what Techopedial (2018) calls intellectual isolation. In the context of social and political opinions, this is often called the polarization effect (Wikipedia contributors. 2018).
more likely look for select things that fit their existing preferences), and
12

---

Under review as a conference paper at ICLR 2021
Filter bubbles be encouraged by algorithms in two main ways. The first is the most commonly described: simply by showing content that is similar to what a user has already searched for, search recommender systems create a positive feedback loop of increasingly-simine similar content (Pariser) 2011 Kayhan 2015). The second way is similar but opposite if the predictions of an algorithm are good for certain group of people, but had for others, the algorithm driving hard-to-predict users away. Then new users to the site will either be turned off' entirely, or see an artificially homogenous community of like-minded peers, phenomena (Shah et al. (2018 call positive externalities.
do better its metrics by
In study 50,000 US-based internet users, Flaxman & Goel (2015) find that two things increase with social media and search engine use: (1) exposure of an individual to opposing or different viewpoints, and (2) mean ideological distance between users. Many studies eite the first result as evidence of the benefits of internet and social media (Robson 2018 Bakshy et al., 2015), but the correlation of exposure with ideological distances demonstrates that exposure is not enough, and might even be counterproductive.
Facebook's own study on filter bubbles results show that the impact of the news feed algorithm on filter bubble "size" (a measure of homogeneity of posts relative to a baseline) is almost as large as the impact of friend group composition (Bakshy et al. 2013). Kayhan (2015) specifically study the role of search engines in confirmation bias, and find that search context and the similarity of results in search engine results both reinforce existing biases and increase the likelihood of future biased searches. Nguyen et al. (2014) similarly study the effect of recommender systems individual users' content diversity, and find that the set of options recommended narrows over time.
Filter bubbles create ideal environment for the spread of fake news: they increase the likelihood of repeat viewings of similar content, and because of the illusory truth effect, that content is more likely
be believed and shared (Pennycook al. 2019 DiFranzo Gloria-Garcial 2017 Pariser 2011) 1). We are not claiming that HI-ADS are entirely or even mostly responsible for these problems, but we do note that they can play a role that is worth addressing.
13

---

Under review as a conference paper at ICLR 2021
incentive-compatible (3 0.5)
#agents-10
#agents-
#agents-1000
1.00
Budddwws AUD oM
0.75
0.50
0.00
1.00
Piverbose)
Buiddeas AWR
0.50
0.25
0.00
0
250
500
750
1000
0
250
500
750
1000
250
500
750
1000
time step
time-step
step
incentive-orthogonal (B 0.0)
#agents=22 10
#agents=100
#agents=1000
1.00
Guiddiaws
0.75
0.50
1.00
0.50
0.25
0.00
250
500
750
1000
250
500
750
1000
250
500
750
1000
time-ster
incentive-opposed (3 = -
#agents=10
#agents=100
#agents=1000
1.00
Guddewss AUD
0.75
0.50
0.25
0.00
1.00
Buiddows AUR
0.75
0.50
0.25
0.00
0
250
500
750
1000
0
250
500
750
1000
0
250
500
750
1000
time-step
time-step
time step
PBT PBT interval
PBT interval
PBT interval 100
Figure 8: Average level of non-myopic (i.e. cooperate) behavior learned by agents in the unit test for HI-ADS. Despite making the inner loop fully myopie 0), population-based training (PBT) can cause HI-ADS, leading agents to choose the cooperat action (top row), context swapping successfully prevents this (bottom row). Columns (from left right) show results for populations of 10, 100, and 1000 learners. In the legend, "interval" refers to the interval (I") of PBT (see Sec. 2.2). Sufficiently large populations and short intervals are necessary PBT to induce nonmyopic behavior.
14

---

Under review as a conference paper at ICLR 2021
EXTRA EXPERIMENTS AND REPRODUCIBILITY DETAILS
9.1 HI-ADS UNIT TEST
We used REINFORCE (Williams 1992) with discount factor 7 as the baseline/IL optimizer. PBT (with default settings, see Section 2.2) is used to tune the learning rate, with reward on the final time-step of the interval as the performance measure for PBT.
Formally, the environment is not 2x2 game (as the original prisoner's dilemma); it's a partially observable Markov Decision Process Aström 1965 Kaclbling et al. 1998):
St, Or 01-1, 0
at fdefect, cooperate]
P(st, ax) = 8(at)
R(s,, (2)) = I(s, cooperate) B I(a = cooperate) - 1/2
where is indicator function, and 3 1/2 parameter controlling the alignment incentives. The initial state is sampled as 80 (dd fect. cooperate). Policies are represented by single valluv parameter 0 (initialized as (O0, 1)) passed through a sigmoid whose output represents P(a = defect).
9.1.1 ALIGNMENT OF INCENTIVES EXPLORATION
This section presents an exploration of the parameter 3, which controls the alignment of incentives in the HI-ADS unit tests (see Table 2).
To clarify the interpretation of experiments, we distinguish between environments in which myopic (defect) vs. nonmyopic (cooperate) incentives are opposed, orthogonal, or compatible. Note that in this unit test myopic behaviour (defection) is what we want to see.
1. Incentive-opposed: Optimal myopic behavior is incompatible with optimal nonmyopic behav- ior (classic prisoner's dilemma; these experiments in the main paper).
2. Inchnnive-orthogonal:: Optimal myopic behavior may may not be optimal nonmyopic behavior.
Incentive-compatibl:: Optimal myopic behavior is necessarily also optimal nonmyopic behav- ior.
We focused on incentive-opposed environment (3 = 11) in the main paper in order to demonstrate that HI-ADS can be powerful enough to change the behavior of the system in an undesirable way. Here we also explore prretive-compatione and incentive ortthova-rrthogonal environments because they provide useful baselines, helping us distinguish a systematic bias towards nonmyopic behavior from other reasons (such as randomness optimization issues) for behavior that does not follow myopically optimal policy.
9.1.2 WORKING THROUGH DETAILED EXAMPLE FOR PBT WITH T =
To help provide intuition on how (mechanistically) PBT could lead to persistent levels of cooperation, we walk through simple example (with no inner loop). Consider PBT with T - and population of 5 deterministic agents AI,..., As playing cooperate and receiving reward of r(A,) = Now suppose A₁ suddenly switches to play defect. Then rí ((A)) = 1/2 the next time-step (while the other agents' reward is still 0), and so PBT's EXPLOIT step will copy A₁ (without loss of generality to A2). On the following sime-step, r(A2) 1/2, and r r(A1)= 1122, PBT will clone Ag to At, and the cycle repeats. Similar reasoning applies for larger populations, and >
9.1.3 Q-LEARNING EXPERIMENT DETAILS
We show that, under certain conditions, Q-learning can learn to (primarily) cooperate, and thus fails the HI-ADS unit test. We estimate Q-values using the sample-average method, which guaranteed to
15

---

Under review as a conference paper at ICLR 2021
Table 2: controls the extent to which myopic and nonmyopic incentives are aligned. Cooperating
B
Environment
incentive-opposed incentive-orthogond. incentive-compatile
yields less reward on the current time-step (myopically detrimental)
does not affect the current reward (myopically indifferent) yields more reward on the current time-step (myopically beneficial)
converge in the fully observed, tabular case (Sutton Barto 1998). The agent follows the e-greedy policy with 0.1. In order to achieve this result, we additionally start the agent off with one synthetic memory where both state and action are defect and therefor R(defect) hard-code the starting state to be cooperate (which normally only happens 50% of the time). Without this kind of an initialization, the agent always learns defect. However, under these conditions, we find that 10/30 agents learned to play cooperate most of the time, with ((cooperate) and Qddefect) both hovering around -0.07, while others learn to always defect, with Q (ccopperate) -0.92 and Q(defect) & -0.45. context swapping, however, prevents majority-coopene behavior from ever emerging, see Figure
-.5, and
9.1.4 Q-LEARNING: FURTHER RESULTS
To give a more representative picture of how often Q-learning fails the unit test, we run a larger set of experiments with Q-learning, results are in Figure TO It's possible that the failure of Q-learning is not persistent, since we have not proved otherwise, but we did run much longer experiments and still observe persistent failure, Figure 9
0.0
0.0
Picooperate) Q-values
0.5
-0.5
1.0
1.0
1.0
0.9
0.8
0.0
50000
50000
50000
time-step
time-step
time-step
time-step
Q(defect)
O(cooperate)
#cooperate/time-step
Figure 9: The same experiments as Figures 6 illustrate the persistence of non-myopic behavior.
run for 50,000 time-steps instead of 3000, to 10
16

---

Under review as a conference paper at ICLR 2021
Picooperatel Q-values
0.0
0.0
0.0
-0.5
-0.5
-0.5
0.5
0.5
0.5
0.5
0.5
2500
0
2500
2500
0
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Occooperate)
#cooperate/time-step
0.0
0.0
Picooperate) Q-values
-0.5
-0.5
-0.5
00.5
-0.5
1.0
1.0
1.0
1.0
1.0
0.5
0.5
0.9
0.9
2500
2500
2500
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Q(cooperate)
#cooperate/time-strr
0.0
0.0
0.0
P(cooperate) Q-values
0.0
-0.5
0.5
0.5
-1
-0.5
0.5
1.0
0.5
0.5
0.5
0.9
0.0
0.0
2500
2500
2500
0
2500
2500
time-step
time-step
time-step
time-step
time-step
Oldefect)
Olcooperate)
#copperatertime-step
0.0
0.0
Picooperate) Q-values
0.0
0
0.0
-0.5
-1
-0.5
1.0
1.0
1.0
1.0
1.0
0.8
0.9
0.9
www
0.0
2500
2500
2500
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
O(cooperate)
#cooperate/time-step
0.0
Q-values
0.0
0.0
0.0
0.5
005
0.5
00.
1.0
P(copprraee
1.0
1.0
1.0
1.0
0.5
0.5
0.0
0.0
0.0
2500
2500
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Occooperate)
#cooperate/time-step
Figure 10: More independent experiments with Q-learning, exactly following Figure 0 Q-learning fails the unit test in total of 10/30 experiments (including those from Figure 6).
17

---

Under review as a conference paper at ICLR 2021
Q-values
0.0
0.0
0.0
0.0
0
-0.5
0.5
0.5
-0.5
-1
Picooperate)
1.0
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
2500
2500
2500
0
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
O(cooperate)
#cooperate/time-ste
0.0
Picooperate) Q-values
0.0
0.0
-0.5
-0.5
-0.5
00.5
1.0
1.0
1.0
1.0
1.0
0.5
0.0
time-step
time-step
time-step
time-step
time-step
Q(defect)
O(cooperate)
#cooperate/time-step
0.0
0.0
0.0
Q-values
0.0
0
-0.
-0.5
Acooperate)
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
2500
2500
2500
0
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Occooperate)
#cooperate/time-step
0.5
Q-values
0.0
0.0
-0.5
0.5
00.5
-1
1.0
1.0
1.0
Pccoperate)
0.5
0.0
2500
2500
2500
0
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Olcooperate)
#cooperate/time-step
Picooperate) Q-values
0
0.5
-1
0
0.0
0
2500
time-step
time-step
time-step
time-step #cooperate/time-step
time-step
Q(defect)
((copperate)
Figure 11: More independent experiments with Q-learning, exactly following Figure 6 except also using context swapping. This leads to 100% success rate on the unit test.
18

---

Under review as a conference paper at ICLR 2021
9.2 CONTENT RECOMMENDATION
9.2.1 I
ENVIRONMENT DETAILS
The evironment has the following components:
1. User type, x²: categorical variable representing different types of users. The content recommender conditions its predictions on the type of the current user.
2. User loyalty, g': the propensity for users of each type to use the platform. User æ' is sampled from a categorical distribution with parameters given by softmax(g").
3. Article type, y': categorical variable (one-hot encoding) representing the type of article selected by the user.
4. User interests, W a matrix whose entries W z.v represent the average interest user of type x have in articles of type y.
At each time step 1, a user x' is sampled from a categorical distribution (based on the loyalty of the different user types), then the recommendation system selects which type of article to present in the top position, and finally, the user selects an article. The goal of the recommendation system is to predict the likelihood that the user would click on each of the available articles, order select the one which is most interesting to the user.
User loyalty for x then changes in accordance with the self-selection effect, increasing or decreasing proportionally to their interest in the top article. The interests of user type x² (represented by column of W also change; in accordance with the illusory truth effect, their interest in the topic of the top article (as chosen by the recommender system) always increases. Overall, this environment an extremely crude representation of reality, but allows us to incorporate both the effects of self-selection (via covariate shift), and the illusory truth effect (via concept shift).
Formally, this environment is similar to a POMDP R. i.e. a POMDP with no reward function, also known as world model (Armstrong & O' O'Rourke, 2017; -adfield-Mneel et al., 2017); the difference is that the learner observes the input before acting and only observes the target after acting. The states, observations, and actions given below.
s' (g' W W,,,,,, ,²,""") opee a', opest = (x", ".y'".
The state transition function is defined by:
g.
g + aa +a,W z1.01
W
w 1412 x''''
Wt z''0" + +a; W = g¹
|w a²
x 1+1 softmax(g 1+1
y softmax w
Where je is the top article as chosen by the recommender, and 0i,, 02 represent the rate of covariate and concept shift (respectively). The update for W merely increases the interest of user type x! in article type ğ', then normalizes the interests for that user type.
9.2.2 REPRODUCIBILITY DETAILS
For these experiments, the recommendation system is a ReLU-MLP with hidden layer of 100 units, trained via supervised learning with SGD (learning rate 0.01) to predict which article user will select. Actions are sampled from the MLP's predictive distribution. We apply PBT without any hyperparameter selection (this amounts to just doing the EXPLOIT step), and an interval of 10, selecting on accuracy. We use a population of 20 learners (whether applying PBT not), and match random seeds for the trials with and without PBT. We initialize g' and W to be the same across the 20 copies of the environment (i.e. the learners start with the same user population), but these values diverge throughout learning. For the environment, we set the number of user and article types both to 10. Initial user loyalties randomly sampled from (0, 0.03), 0n 0.03, and 02 0.003.
19

---

Under review as a conference paper at ICLR 2021
9.2.3 DETAILS OF EVALUATION
We measure concept shift (change in P(y[x) as the cosine distance between each user types' initial and current interest vectors. And we measure covariate shift (change in P(x)) as the KL-divergence between the current and initial user distributions, parametrized by g' and g' respectively. Results are presented in / (main text). In Figure [2] we additionally plot concept shift and covariate shift as function of accuracy. We observe that for both types of ADS, at low levels of accuracy PBT actually causes less shift than occur in baseline agents; HI-ADS are only observed for accuracies above 60%. This suggests that only relatively strong performers are able pick up on the HI-ADS revealed by PBT (Fig. 12
2355
0.51
22360
2,205
PXV0 (2V0)ccmsmel (cosinel
0.03
7,2000
2.000
2.2025
2,2000
06
Amount of auto-induced covariate shift (left) and auto-induced concept shift (right) as function of performance (aceuracy) averaged over all trials, learners, and time-steps. Only relatively strong learners (those which achieve accuracy > 60%) exhibit HI-ADS.
Figure 12:
9.2.4 CONTEXT SWAPPING IN CONTENT RECOMMENDATION
We believe context swapping is not appropriate for the content recommendation environment, since when the environments diverge, optimal behavior may differ across environments. Nevertheless, we experiments with it for completeness. The main effect appears be to hamper learning when PBT is not used, see Figure [3. Notably, it does not appear to significantly influence the rate or extent of ADS when combined with PBT.
9.2.5 EXPLORATION OF ENVIRONMENT PARAMETERS
In Figure 14 examine the effect of the rate-of-change parameters (01, 0:2) of the content recommendation environment on the results provided in the paper. As noted there, our results are qualitatively consistent long as (I) the initial user distribution is approximately uniform, and (2) the covariate shift rate (a1) is faster than the concept shift rate (a2). These distributions are updated by different mechanisms, and are not directly comparable. Concept shift changes the task more radically, requiring a learner to change its predictions, rather than just become accurate on wider range of inputs. We conjecture that changes in P (z) must therefore be kept smooth enough for the outer loop to have pressure to capitalize on HI-ADS.
20

---

Under review as a conference paper at ICLR 2021
Accuracy of click prediction
Change in P(Y[X) (user interests)
Change in P(X) (user base)
coline distance from original interests
2,5
0.05
div. from original distribution
2.0
Accuracy
15
1.0
0.02
0.01
0.5
0.00
0000
0.0
timesstep
time-step
timesstep
PBT
no PBT
PBT, env. swapping
no PBT, env. swapping
cosine distance from ongiiaal interests
Change P(X) (user base)
Change in P(Y[X) (user interests)
original distribution
0.05
2.305
0.04
2.300
0.03
2.295
2.290
0.01
2.285
0.00
0.6
0.8
1.0
0.00
0.25
0.50
0.75
1.00
accuracy
accuracy
PBT
no PBT
PBT, env. swapping
no PBT, env. swapping
Figure 13: environment.
Context swapping doesn't have the desired effect in the content recommendation
21

---

Under review as a conference paper at ICLR 2021
01
0.01 02 0.001
a1 0.1 0/2 0.001
Accuracy of dlick prediction
Accuracy of dlick prediction
1.0
1.0
05
05
400
1200
2800
2000
400
1200
28:00
2000
time-step
Change interests)
Change interests)
0.2
Simo stup
Eino slup
Change in PXX) (user base!
Change in P(XI (user basel
0-
400
800
1200
1800
2005
400
000
1200
1600
2000
no
no
C11 0.01 0g
0.01
0.1 02
0.01
Accuracy of prediction
Accuracy ciick prodiction
1.0
1.0
03-
0.5
400
B05
1200
1600
2000
400
920
1200
1000
2000
Steg
time stop
Change P(Y[K) (user interests)
Change PIYIKI interests)
0075
0.000
003
0025
BD0
timestte
stop
Change P(X) [user base!
Change (user base)
2-
0-
400
800
1200
1600
2000
400
800
1230
1000
2000
0:1 0.01 02
0.1
0.1 02
0.1
Accuracy click prediction
Accuracy of prediction
2.0
1.0
05-
0C
400
820
1200
1600
2000
400
RD0
1200
1600
2000
Iimestop
tme-step
Change in P(YI) (user interests)
Change P(YP)) [user interests)
0.
0.15
02
005
Imeestep
tme-step
Change PIX) basc)
Change in POX) (user base!
2-
0-
400
800
1200
1600
2000
400
800
1200
1600
2000
PET
Figure 14: Content recommendation results for different values of (1), Q2.
22

---

Under review as a conference paper at ICLR 2021
10 OFFLINE Q-LEARNING CAN REVEAL INCENTIVES FOR ADS
First, recall that this unit test is a POMDP, and the state is not observed. Since there are only 2 possible actions, a policy is defined by a single parameter 0 p( (coopperate). Now, the state distribution is P(s = cooperate) = 0 (ignoring the first state, which is appropriate in the limit of infinite data). More specifically, the probability of each state-action combination are as follows:
Suppose we have a dataset of N examples generated by following a fixed policy.
Q(C) = |s C, R( C, a = C) + D, C ( D C) C)
(1)
aa a= C C|
NP C, C, = B.
D
DDa=
NN
P(s C, a C)R(s C, a C) P(s C)R(=)R((= D,
P(a=
RR(s C, a C) + 011 0)H(s D,a CC)
(4)
AR(s C C) + (1 (1- R(s D.a C)
(5)
=0(0) - (1 - (-
(6)
0 -11
Q(D) s= D
|a= D
P(s
= C,a = D) P +)
P(s==DDa= D)R(s -),(s=DDR=>>11 D,a=
P(a=D)
P(s= C D) + P(s = D)R(s D,a=
(10)
-0(1/2) + (1 (1-0))-11/1)
(11)
=1/2(20- 1)
(12)
102
(13)
So that Q(D) QCC), regardless of 6.
Now, suppose instead that we have N examples from each of 2 different policies (given by parameters 8, and 02) operating in different environments. Intuitively, this sort of data might arise in practice from "A/B testing", where 2 different users have been assigned to different policies in order to compare the policies' performance. We now DC to represent D, a C, etc.
Q0) (CC|R(CC) + DC(R(DC)
(14)
C
N(P*) (CC) + Pda (CC))R(CC)++(PP))))
+ (D
(15)
N(POL (C)+ p0: (C)(
(CC) + poa (CC)) + (Po) (DC) p6d (DC))R(DC)
(16)
(P): + p8: (C))
(0f + (1++8) + 01) + 02 R(D)
(17)
01 + 02
(1(1 - 01) + 02(1
02)
(18)
01 + 02
09 0 + 03 02
(19)
01 + 02
(20)
23

---

Under review as a conference paper at ICLR 2021
Q (D) (DIR(CD) + (DDIR(DD)
(21)
[D] (P*) + po (CD)) (Po) (DD) pm (DD))R(DD)
(22)
(P) + 2 (D))
1/2(01 81) + 02 02)) 1/2((1 01) -1-82)) 02)²)
(23)
2 01 02
(20) 1)(1 01) + (202 1)(1 - 02)
(24)
4 20 202
Now, in Figure 15 we see that Q(C) > Q(D) when one of the policies cooperates with high
probability, and the other defects with high probability. Intuitively, the result of pooling data from such policies is very similar to collecting data from an e-greedy policy trained online (as in Figure 6).
10
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
10
0
Figure 15: Offline Q-learning can also reveal HI-ADS, when pooling data from different (policy, environment) pairs. Yellow regions represent policy pairs for which Q(C) > Q(D), resulting in non-myopic behavior.
24


Reviews:
[{"review_id": "eTOYhiPBY2Q", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses a phenomenon where machine-learned models may influence user behaviors in future iterations, creating self-selection effects such as filter bubbles or propagation of fake news. The paper calls these effects auto-induced distribution shift (ADS) and argues that a specific meta-learning algorithm PBT manipulates users instead of maximizing rewards. To illustrate the ideas, the paper introduces a few simulation environments where user behaviors may change when certain prediction events happen.\n\nDespite an interesting motivation, the paper is quite vague in the proposed methods. For example,\n1. I am not able to associate ADS with the specific meta-learning algorithm called PBT. Self-selection is a universal phenomenon present in many types of algorithms. It can be corrected by any learners that focus on the detection of distribution shifts, e.g., via inverse propensity scoring, or under ignorability assumptions. On the other hand, PBT is just an algorithm with automated hyperparameter search. What can we learn by connecting ADS to PBT?\n2. Why are reinforcement learning / supervised learning free from incentive / ADS issues? For the reasons above, I believe these claims are unrelated and wrong.\n3. Is learning conducted online or offline? Would user states be observable or unobservable? If offline, how would the learner foresee the shifts in user behaviors after online feedback loop? If online and observable, shouldn't we focus on modeling user state transitions? If online and unobservable, what are the belief states?\n\nAdditionally, the experiments are far from being practical. While Experiment 1 makes sense that by sacrificing an immediate reward, the user gets \"manipulated\" into a state of more predictability, I am not seeing how Experiment 2 makes practical sense. In this example, the environment is set as a modified prisoner's dilemma. The paper assumes \"defect\" to be a better solution, but in this specific case, it seems that cooperation seems to be the true user utility, despite the user gets \"manipulated\" by non-myopic RL algorithms.\n\nThe paper also lacks general clarity. I am not sure how content swapping works in Section 4.2. The details about Q-learning should be reorganized. Section 5.2 is best presented with equations or diagrams. Overall, the authors should think about how to eliminate the unrelated claims and simplify the paper to include just one proposed method, which solves a wide class of interesting problems at the same time.", "extra_scores": {}, "extra_text": {}}, {"review_id": "bEHJMg1WsOl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Abstract: The paper highlights an interesting problem of learning dynamics, where a learning system has the incentives to change it’s future input in order to increase its performance. This is not always problematic, but it can, at times, lead to perverse incentives for the system, such as under-performing on users with complex profiles in the present, in order to remove them from the future testing sets. The authors propose a set of unit tests to detect this issue and a way to solve the problem for the PBS metalearning frameworks. The experiments on toy/simulated datasets show the relevance of the approach.\n\n\nPros: \n- Clarity: The paper is quite well-written and the explanations are clear. I think the problem that the paper aims to address is real and of practical value.\n- Experimental design: The experimental section, is very clear and supports the claims of the paper\n\nCons:\n- Significance/Impact: I find the scope of the paper to be limited to Population Based Training, which as far as I can tell, its not a well-established training method in real-world applications, especially recommendation. I would like to see the expansion of the same ideas to general batch learning from bandit feedback which is as far as I can tell the most established setup for RW ML decision systems.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YtLfDWvwp9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces the concept of auto-induced distributional shift (ADS), and argues that some meta learning and reinforcement learning algorithms have the incentives to change the distribution so that the problem is easier to solve. The paper presents unit tests to detect hidden incentives for auto-induced distributional shift (HI-ADS) and also proposes context swapping to reduce the distributional shift. Experiments on Population-Based Training (PBT) show that PBT reveals HI-ADS in unit tests and context swapping mitigates the distributional shift for PBT.\n\nPros: \n1) The concept of ADS is interesting. Distributional shift is not a new phenomenon especially for reinforcement learning. As mentioned in the paper some existing works try to exploit the distributional shift. And if distributional shift is harmful then algorithm could use a better reward function to mitigate the effects. But this concept is indeed not formally defined and studied in previous works.\n\n2) The results of unit tests reveal hidden incentives of PBT under two toy environments. \n\nI have the following concerns and questions:\n\n1) How to measure HI-ADS is unclear. From the definition of HI-ADS in Section 4.1, it seems that we should evaluate by whether \"the learner would not learn to perform the incentivized behaviors at higher than chance levels\".  In the unit test, the incentivized behaviors are clearly defined for the two toy environments, but the paper does not discuss how to find / define incentivized behaviors for a general environment. This directly leads to my second concern.\n\n2) How to generalize the unit test? The unit test only tells whether an algorithm reveals ADS in the two toy environments. Even if a meta learning algorithm passes the unit test, it is not guaranteed that such algorithm won't exhibits ADS in other application scenarios. Can we get more from the unit test? In real world applications if we do not have a perfect understanding of the environment that what kind of behaviors are encouraged by incentives, then how to evaluate HI-ADS?\n\n3) It is not clear whether PBT algorithm itself or all meta learning algorithms would exhibit HI-ADS. The authors use PBT as an example to illustrate HI-ADS, but it is unclear whether this observation suggests other meta learning algorithms are also vulnerable to ADS or that we should test them individually using the unit tests. \n\n4) Why would context swapping mitigate the distributional shift? It is not clearly explained. My rough understanding is after switching the environment, the learners are using misspecified models and won't fully exploit current estimation.  Does context swapping help with content recommendation task? Is this a general approach for other environments instead of the toy unit test setting?\n\n\nOther comments:\n\n1) Figure 5 and 6 are too small to read. \n\n2) Appendix is missing, which is supposed to contain additional details on experiments according to the main content.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ApMHIsuYOvH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to study auto-induced distribution shift (ADS), a phenomenon where the machine learning model itself may change its own data distribution via its decisions or actions. The paper seeks to understand, in situations where ADS is undesirable, whether learning algorithms are privy to, and pursue, incentives for learning models which take advantage of ADS.\n\nPros:\n\n+ The paper proposes an interesting and promising direction for study.\n+ Simple examples serve to illustrate the concepts outlined in the paper.\n\nCons:\n\n- Both expositionally and experimentally, there isn't a compelling argument made in terms of when ADS shows up in real problems.\n- There is limited insight as to how the problem of interest may be addressed methodologically.\n\nWeighing these pros and cons, I am inclined to reject this paper. More details are provided below.\n\n\nQuality\n---\n\nPerhaps what detracts from this work the most is the lack of rigorous evaluation on real world problems. It is not difficult to imagine that ADS may indeed be a problem that must be dealt with in certain applications, but it is also unsatisfying to leave this to the imagination. All of the experiments presented are toy, and though toy experiments are certainly valuable for gaining insights into the problem, they should be accompanied also by larger studies that corroborate the story.\n\nThe content recommendation experiment seems promising and could perhaps be built out into a larger scale experiment. But this raises additional questions about why the proposed method of context swapping is not effective at handling this situation. In general, the context swapping approach seems rather ad hoc, and might there be potential downsides regarding tradeoffs in model performance? Further investigation into this or other mitigation strategies would also significantly strengthen the work.\n\nClarity\n---\n\nThe paper is generally well organized and well written. I appreciate the hierarchy of definitions as well as the discussion of how certain frameworks such as meta learning can exacerbate the problem being studied.\n\nOriginality\n---\n\nI am not an expert in this area, but the paper seems to propose an original problem to study, and discussion of related work seems relatively well researched and well written.\n\nSignificance\n---\n\nRelating back to my previous comments, the exposition needs to do a better job at convincing the reader that the problem is important. Further discussion into some of the prior works mentioned in the related work section would help in this regard. Is it possible to characterize prior work into the taxonomy and framework described? Might it even be possible to design (or borrow) an experiment based on this prior work?\n\nIn my opinion, the current exposition is not enough to convince the reader that the problem is important. The running content recommendation example seems hypothetical, and a real world system would likely, perhaps even obviously, disincentivize such behavior from a model that drives users away. This indeed may be a hand designed solution, which as the authors correctly point out may not scale to cover all instances of this problem, but it is up to the authors to argue this more convincingly.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a thought-provoking paper which describes a significant problem that plausibly occurs in deployed ML/RL models.
The paper is clearly written, describing claims using examples and developing small unit-tests to probe models.
However, as the reviews and discussion show,  the exposition should be substantially re-worked so that the core contributions are more understandable -- the core message in the revised manuscript is still very nuanced and easy to mis-understand.

Let's say we train an ML model using supervised learning to minimize a loss function on a dataset. Several models may have near-optimal loss as measured on a validation set -- a learning algorithm is free to return any one of them. Now in a deployed system, these ML models are not merely generating passive predictions; these predictions are driving system operation and potentially influencing future states/contexts/inputs that the model will be invoked on. It is well known that supervised learning makes an iid assumption between training and deployment which is violated in this setting -- that is not the main point of this paper. Consider again the set of models with near-optimal loss. Some of them, when deployed, may cause the distribution mismatch between training and deployment to be miniscule, while other models may introduce a vast mismatch. We may choose a learning algorithm which just so happens to pick models from the former category; and we may conclude that feedback effects induced by the ML model are not substantial. We then change some unrelated detail in the learning algorithm (but not the objective, datasets, validation criteria, etc.) which just so happens to pick models from the latter category and suddenly witness a large distribution shift. What happened? And could we have developed tests to detect that our learning algorithms have these tendencies? The paper attempts to articulate such questions, and design the first step in answering them.

Moving to RL, where we routinely consider distribution shifts in states visited by different policies, does not fundamentally fix all these issues because the reward function is typically an engineered proxy to elicit desired behavior -- and we may again find that some RL algorithms have a tendency to find reward-maximizing policies that exploit gaps in reward specification as opposed to following intended behavior.

The core question studied in this paper, scoped to the supervised learning setting, is very related to that of strategic classification (see e.g., https://arxiv.org/pdf/1910.10362.pdf Strategic Classification is Causal Modeling in Disguise). The following sketch is inspired by that literature.

We might hope to augment the training objective of ML/myopic RL/strategic RL to address the Auto-induced distribution shift problem as follows.
[Supervised learning for content recommendation] Let the training/validation data distribution be D. Assume for now that there is no exogenous factor in the environment that causes any distribution shifts in deployment -- so, the only shift is due to feedback effects from the predictions made by the model. For an ML model f, let the corresponding recommendation policy be pi_f, and let the long-term distribution of data seen from user-interactions with pi_f be D[pi_f]. Then, what we want is: f* = argmin_F Expectation over D [ L(f) ] subject to constraint that D ~= D[pi_f]. 
For a contextual bandit/myopic RL formulation of the problem, we could similarly constrain the learning problem as pi* = argmax_Pi Expectation over D [ Reward of pi] subject to constraint that D \approx D[pi].
Essentially, both supervised ML and contextual bandit algorithms are assuming that context distribution is unchanged -- so let us enforce that the context distribution is indeed unchanged as a consequence of the policy's actions.
It is unclear how to generalize this kind of thinking to situations where environmental changes also contribute to distribution shift.  The authors call out precisely this flaw using the cryptic comment -- 'not trying to change X' is not the same as 'trying to not change X'. The formulation above does 'trying to not change X', but that is an insufficient band-aid in situations when environment changes X. It's also unclear how one might estimate D[pi] or D[pi_f] and appropriately constrain the learning algorithm -- but these are all interesting questions to study.

The paper in its current form is asking an important question. In supervised learning, the desired solution might actually coincide with strategic classification solution concepts. The paper may be asking a generalization of the phenomenon for myopic RL and RL. It may spark interesting discussions and follow-up work, but is not yet mature beyond a workshop poster.
Generalizing the unit-tests, articulating the scope of situations where context-swapping may be a useful strategy, and even formalizing the problem and desired goal (as attempted above for the content recommendation example) will substantially strengthen the paper.
Decision: Reject