Revealing the Incentive to Cause Distributional Shift
Authors: David Krueger
Tegan Maharaj
Jan Leike
Venue: ICLR 2022 Submitted (2022)
Keywords: alignment
incentives
unit testing
distributional shift
content recommendation
myopic reinforcement learning

Abstract: Decisions made by machine learning systems have increasing influence on the world, yet it is common for machine learning algorithms to assume that no such influence exists. An example is the use of the i.i.d. assumption in content recommendation: In fact, the (choice of) content displayed can change users’ perceptions and preferences, or even drive them away, causing a shift in the distribution of users. We introduce the term auto-induced distributional shift (ADS) to describe the phenomenon of an algorithm causing change in the distribution of its own inputs. Leveraging ADS can be a means of increasing performance. But this is not always desirable, since performance metrics often underspecify what type of behaviour is desirable.  When real-world conditions violate assumptions (such as i.i.d. data), this underspecification can result in unexpected behaviour. To diagnose such issues, we introduce the approach of unit tests for incentives:  simple environments designed to show whether an algorithm will hide or reveal incentives to achieve performance via certain means (in our case, via ADS). We use these unit tests to demonstrate that changes to the learning algorithm (e.g. introducing meta-learning) can cause previously hidden incentives to be revealed, resulting in qualitatively different behaviour despite no change in performance metric. We further introduce a toy environment for modelling real-world issues with ADS in content recommendation, where we demonstrate that strong meta-learners achieve gains in performance via ADS. These experiments confirm that the unit tests work – an algorithm’s failure of the unit test correctly diagnoses its propensity to reveal incentives for ADS.

Under review as a conference paper at ICLR 2022
REVEALING THE INCENTIVE TO CAUSE DISTRIBUTIONAL SHIFT
Anonymous authors Paper under double-blind review
ABSTRACT
Decisions made by machine learning systems have increasing influence on the world, yet it is common for machine learning algorithms to assume that no such influence exists. An example the use of the i.idd assumption in content recom- mendation. In fact, the (choice of) content displayed can change users' pereep- tions and preferences, or even drive them away, causing a shift in the distribution of users. We introduce the term auto-induced distributional shift (ADS) to de- scribe the phenomenon of am algorithm causing a change in the distribution of its own inputs. Whether it's desirable (or not) for an algorithm to cause ADS is not captured by its performance metric - metrics are an incomplete specification desired behaviour. When real-world conditions violate assumptions, this under- specification can result in unexpected behaviour or 'gaming'. To diagnose when this happens, we introduce the approach of unit tests for incentives: simple en- vironments designed to show whether an algorithm will hide or reveal incentives to achieve performance via certain means (in our case, via ADS). We use these unit tests to demonstrate that changes to the learning algorithm (e.g. introducing meta-learning) can cause previously hidden incentives to revealed, resulting a complete change in behaviour despite no change in performance metric. We further introduce toy environment for modelling real- weal-world issues with ADS content recommendation, where we demonstrate that strong meta-learners achieve gains in performance via ADS. These experiments confirm that the unit tests work an algorithm's 's failure of the unit test correctly diagnoses its propensity to reveal incentives for ADS.
INTRODUCTION
Consider a content recommendation system whose performance is measured by accuracy in pre- dicting what users will click. This system can achieve better performance by either D: Making better predictions, or 2: Changing the distribution of users such that predictions are easier to make. We propose the term auto-induced distributional shift (ADS) to describe this latter kind of distributional shift, caused by the algorithm's 's own predictions or behaviour (Figure IT]. ADS are not inherently bad; often they desirable. But unexpected ADS can lead unintended behavior. While it is common machine learning (ML) to assume (e.g. via the i.i.d. assumption) that ADS will not occur, ADS are inevitable in many real-world applications. Thus it is important to under- stand how ML algorithms behave when such assumptions are violated, i.c. in the actual scenario they will encounter during training this is the motivation of our work.
In many cases, including news recommendation, we would consider 2 a form of specification gaming (Krakovna et al., 2020 the algorithm changed the task rather than solving as intended. We care which means the algorithm used to solve the problem - ! vs. 2 but we only told it about the ends, so it didn't 't know not to 'cheat'. This is an example of a specification problem (Leike et al., 2017] Ortega et al. 2018): problem which arises from discrepancy between the performance metric (maximize accuracy) and "what we really meant" (maximize accuracy only via (). which is difficult to encode as performance metric. Ideally, we'd like to quantify the desirability of all possible means, e.g. assign appropriate rewards to all potential strategies and side-effects, but this is intractable for real-world settings. Using human feedback to learn reward functions which account for such impacts is a promising approach to specifying desired behavior (Leike et al. 2018 Christiano et al. 2017). But the same issue can arise whenever human feedback

---

Under review as a conference paper at ICLR 2022
08
06
06
84
04
00
-000
200
User typd
-200
200
User 530
Time
Figure 1: Distributions of users over time. Left: A distribution which remains constant over time, following the i.id assumption. Right: Auto-induced Distributional Shift (ADS) results in a change in the distribution of users in content recommendation environment. (see Section 5.2 for details).
is used in training: a means of improving performance could be to alter human preferences, making them easier to satisfy. Thus this work, we pursue a complementary approach: managing learners' incentives.
A learner has an incentive behave in certain way when doing so can increase performance (e.g. accuracy reward). Informally, we say an incentive is hidden when the learner behaves as if were not present. But we note that changes the learning algorithm or training regime could cause previously hidden incentives to be revealed, resulting in unexpected and potentially undesirable behaviour. Managing incentives (e.g. controlling which incentives are hidden/ revealed) can allow algorithm designers to disincentivize broad classes of strategies (such as any that rely on manipulating human preferences) without knowing their exact instantiation.
Our goal in this work is to provide insight and practical tools for understanding and managing learners' incentives, via unit tests for incentives. We present unit tests for diagnosing incentives for ADS in both supervised learning (SL) and reinforcement learning (RL). The unit tests both have two means by which the learner can improve performance: one which creates ADS and one which does not. The intended method of improving performance is one that does induce ADS; the other is hidden and we want it to remain hidden. A learner fails the unit test if it pursues the incentive increase performance via ADS. In both the RL and SL unit tests, algorithms (c.g. minibatch SGD) pass the test, but introducing an outer-loop of meta-learning (e.g. Population- Based Training (PBT) (Jaderberg al. 2017)) can lead to high levels of failure.
find that 'vanilla' learning
Our contributions include:
1. Defining Auto-induced Distributional Shift (ADS) and identifying issues that can arise from learners pursuing incentives for ADS in myopic reinforcement learning or online supervised learning problems.
2. Creating unit tests that can determine which learning algorithms are liable to pursue incen- tives for ADS in these settings.
3. Using these unit that affect their tendency to pursue incentives for ADS.
experimentally confirm qualitative features of learning algorithms
4. Constructing a novel synthetic content recommendation environment that illustrates social problems associated with ADS, and experimentally validating that our unit tests are predic- tive of learning algorithms' behavior in this more complex environment.
5. Proposing mitigation strategy called context swapping that can effectively hide incen- tives for ADS.
Broadly speaking, our experiments demonstrate that performance metrics are incomplete specifica- tions of which behavior is desired, and that we must consider other algorithmic choices as part of the specification process. In particular, considering which incentives revealed by different learning algorithms provides a natural way of specifying which means achieving high performance are acceptable.
2

---

Under review as a conference paper at ICLR 2022
2 BACKGROUND
2.1 META-LEARNING AND POPULATION BASED TRAINING
Meta-learning is the use of machine learning techniques to learn machine learning algorithms. This involves running multiple training scenarios in an inner loop (IL), while an outer loop (OL) uses the outcomes of the inner loop(s) as data-points from which to learn which learning algorithms are most effective (Metz et al. 2019). The number of IL steps per OL step is called the interval.
Population-based training (PBT) (Jaderberg et al., 2017) is meta-learning algorithm that trains multiple learners Li,.., Ln in parallel, after each interval (I" steps of IL) applying an evolutionary OL step which consists of: (1) Evaluate the performance of each learner, (2) Replace both param- eters and hyperparameters of 20% lowest-performing learners with copies of those from the 20% high-performing learners (EXPLOIT). (3) Randomly perturb the hyperparameters (but not the pa- rameters) of all learners (EXPLORE). Two distinctive features of PBT are notable because they give the OL more control than many other meta-learning algorithms over the learning process. First, PBT applies optimization to parameters, not just hyperparameters; this means the OL can directly select for parameters which lead to ADS, instead of only being able influence parameter values via hyperparameters. Second, PBT performs multiple OL steps per training run.
2.2 DISTRIBUTIONAL SHIFT AND CONTENT RECOMMENDATION
In general, distributional shift refers to change of the data distribution over time. In supervised learning with data x and labels this can be more specifically described as dataset shift: change in the joint distribution of P(x, [x. y) between the training and test sets Mloreno-Torre et al. 2012] Quionero- Duindelo-Cannela et al. 2009). As identified by Moreno-Toree Torres et al. (2012), two common kinds of shift are: (1) Covariate shift: changing P(x). In content recommendation, this corresponds to
changing the user base of the recommendation system. For instance, media outlet which publishes inflammatory content may appeal to users with extreme views while alicnating more moderate users. This self-selection effeect (Kayhan| 2015) may appear to recommendation system
increase in
performance, leading to a feedback effect, as previously noted by [Shah et al. (2018). This type of feedback effect has been identified as contributing filter bubbles and radicalization (Pariser [011]] Kayhan 2015). (2) Concept shift: changing P(y)x). In content recommendation. this corresponds to changing a given user's interest in different kinds of content. For example, exposure fake news story has been shown to increase the perceived accuracy of (and thus presumably future interest in) the content, an example of the illusory truth effect (Pennycook al. 2019). For further details on such effects in content recommendation, see Appendix
AUTO-INDUCED DISTRIBUTION SHIFT (ADS)
Auto-induced distribution shift (ADS) is distributional shift caused by an algorithm's 's behaviour. This is in contrast to distributional shift which would happen even if the learner were not present- e.g. for a crash-pred crush-prediction algorithm trained on data from the summer, encountering snowy roads is an example of distributional shift, but not auto-induced distributional shift (ADS).
We emphasize that ADS are not inherently bad or good; often ADS can even be desirable: consider the crash-prediction algorithm. If it works well, such a system will help drivers avoid collisions, thus making self-refuting predictions which result in ADS. What separates desirable and undesirable ADS? The collision-alers system alters its data distribution in a way that aligned with the goal of fewer collisions, whereas the news manipulation results in changes that are misaligned with the goal of better predicting existing users' interests (Leike et al. 2018).
In reinforcement learning (RL), ADS are typically encouraged as means to increase performance. On the other hand, in supervised learning (SL), the i.i.d. assumption precludes ADS in theory. In practice, however, the possibility of using ADS to increase performance (and thus incentive to do so) often remains. For instance, this occurs in online learning. In our experiments, we explicitly model such situations where i.i.d. assumptions are violated: We study the behavior of SL and myopic RL algorithms, in environments designed to include incentives for ADS, in order to understand when incentives are effectively hidden. Fig. contrasts these settings with typical RI. and SI..
3

---

Under review as a conference paper at ICLR 2022
U2
(a) RL: Incentives for (b) SL with i.i.d. data: In-
(c) Myopic RL: Incentives (d) SL with ADS: Incen- for ADS present and tives for ADS present pursuing them is undesir- and pursuing them is un- desirable
ADS present and pur- centives for ADS ab- suing them is desirable sent
able
Figure 2: In the widely studied problems of (a) reinforcement learning (RL) with state s, action 8, reward tuples, and (b) i.i.d. supervised learning (SL) with inputs I, predictions ý and loss I, there are no issues of undesirable incentives for ADS. We focus on cases where there are incentives present which the learner is not meant to pursue (c,d). Lines show paths of influence. The learner may have incentives to influence any nodes descending from its action, A, or prediction, ý. Which incentives are undesirable (orange) or desirable (cyan) for the learner to pursue is context-dependent.
4 INCENTIVES
For our study of incentives, we use the following terminology: an incentive for a behavior (e.g. an action, classification, etc.) is present (not absent) to the extent that the behaviour will increase performance (e.g. reward, accuracy, etc.) (Everitt & Hutter 2019). This incentive is revealed to (not hidden from) learner if would, at higher than chance levels, learn perform the behavior given sufficient capacity and training experience. The incentive is pursued (not eschewed) by a learner if actually performs the incentivized behaviour, Note even when an incentive is revealed, may not be pursued, e.g. due to limited capacity and/or data, or simply chance.
For example, in content recommendation, the incentive drive users away is present some user types are easier to predict than others. But this incentive may be hidden from the learner by using a myopic algorithm, e.g. one that does not see the effects of its actions on the distribution of users. The incentive might instead be revealed to the outer loop of a meta-learning algorithm like PBT, which does see the effects of learner's actions.
Even when this incentive is revealed, however, it might not end up being pursued. For example, this could happen if predicting which recommendations will drive away users too difficult a learning problem, or if the incentive to do so is dominated by other incentives (e.g. change individual users' interests, or improve accuracy of predictions). In general, it may be difficult to determine empirically which incentives are revealed, because failure to pursue incentive can be due to limited capacity, insufficient training, and/or random chance. To address this challenge, we devise extremely simple environments ('unit tests'), ), where
be confident that revealed incentives will be pursued.
Hiding incentives can be an effective method influencing learner behavior. For example, hiding the incentive manipulate users from content recommendation algorithm could prevent from influencing users in way they would not endorse. However, if machine learning practitioners are aware that incentives are present, or that properties of the learning algorithm are hiding them, then seemingly innocuous changes to the learning algorithm may lead to significant unexpected changes in behavior.
Hiding incentives for ADS may seem counter-intuitive and counter- pronucc-produttiveece in the context of rein- forcement learning (RL., where moving towards high-reward states is typically desirable. However, for real-world applications of RI, the ultimate goal is not system that achieves high reward, but rather one that behaves according to the designer's intentions. And as we discussed, can be in- tractable to design reward functions that perfectly specify intended behavior. Moreover, substantial real-world issues could result from improper management of learners' incentives. Examples include tampering with human- gennrrttd reward signals (Everitt & Hutter 2018) (c.g. selecting news arti- cles to manipulate users), and making "self-fuliiling prophecies" (c.g. driving up an asset's 's value by publicly predicting its value will increase (Armstrong & O' Rorke 2017)).
4

---

Under review as a conference paper at ICLR 2022
Hiding incentives for ADS via Context Swapping We propose peppeeeeeeee technique called context swap- ping that can hide incentives for ADS that might otherwise be revealed by the use of meta-learning or other algorithmic choices. The technique trains N learners in parallel, and (e.g. deterministically) shuffles the learners through N different copies of the same (or similar) environments. When N is larger than the interval of the OL optimizer, each learner inhabits each copy for at most single time-step before an OL step is applied. This can hide incentives for ADS in practice, see Sec. 5.1.1
env1
A1
A1
A1
env1
A1
A2
A3
env2
A2
A2
A2
env2
A2
A3
A1
(a) env3
A3
A3
A3
(b) env3
A3
A1
A2
Figure 3: (a) No context swapping (b) Context swapping rotates learners through different environ- ments. This removes the incentive for a learner to "invest" in a given environment, since it will be swapped out of that context later and not be able to reap the benefits of its investment.
EXPERIMENTS
In Section 5.1 we introduce unit tests that determine whether incentives for ADS are revealed. Our experiments show that you can have a learner which behaves as intended, and just by introduc- ing meta-learning (e.g. PBT), without changing the performance metric (e.g. loss or rewards), the learner's ' behavior can change completely. We also show that context swapping is an effective mit- igation technique in these environments. As we demonstrate, the unit tests can be used to compare learning algorithms and diagnose their propensity to reveal incentives.
we model a content recommendation system. The goal of these experiments is to demonstrate how revealed incentives for ADS could create issues for real-world content recommen- dation systems such as news feeds. They also validate the usefulness of the unit tests: algorithms that failed the unit tests also reveal incentives for ADS in this setting. We emphasize that ADS takes place in this environment by construction. The point of our experiments is that meta-learning can increase the rate and/or extent of ADS, by revealing this incentive. We find that context swapping is not effective in this environment, highlighting the need for alternative mitigation strategies.
In Section 5.2
5.1 ADS INCENTIVE UNIT TEST (MYOPIC RL)
This unit test a POMDP (Kaelbling et al.) 1998) inspired by the prisoner's dilemma (Prisner 2014). where agent plays each round against its past self, details in Appendix C.I.I The reward function is presented in Table I An agent in this environment has a long-term, non-myopic, incentive to cooperas rate (with its future self), but current-time-seep. myopie, incentive defectt (from its future self), The unit test evaluates whether learning algorithm reveals the non non-myopic incentive, even when the agent is meant to optimize for the present reward only (i.e. uses discount rate ? = 0). While this may seem like an easy "brute-force" way to hide incentives for ADS, we show it is in fact non-trivial to implement. Naively, we'd d expect the non-myopic incentive be hidden from an agent with y = O. and for the agent to consistently de fect; learning algorithms that do so pass the test. But some learning algorithms fail the unit test, revealing the incentive for the agent to coopera ra te with its future self. We create a similar unit test for supervised learning, and find similar results, detailed in Appendix
C.1.I
Rewards for the RL unit test. Note that the myopic de fect action always increases reward the current time-step. but decreases reward at the next time-step the incentive is hidden from the point of view of a myopic learner. A learner 'fails' the unit test if the hidden incentive to cooperate revealed, i.e. if we see more cooperate (C) actions than de fect. (D).
Table 1:
04
D
4
C
31 a:-1 = D
-1/2
$1 = 41-1 = C
1/2
5

---

Under review as a conference paper at ICLR 2022
5.1.1 MYOPIC RL UNIT TEST EXPERIMENTAL RESULTS AND DISCUSSION
We first show that agents trained with PBT fail the unit tests more often than "vanilla" algorithms which do not use meta-learning. Policies are represented by a single real-valued parameter 0 (initial- ized as A ~ (O. 1)) passed through a sigmoid whose output represents P(ay defect). We use REINFORCE (Williams 1992) with discount factor
0 as the baseline/IL optimizer. PBT (with default settings, see Section 2.2) is used to tune the learning rate, with reward on the final time-step of the interval as the performance measure for PBT. We initialize the learning rate log-uniformly between 0.01 and 1.0 for all experiments (whether using PBT or not).
We expect and confirm that the following two factors lead to higher rates of unit test fail- ure: (1) Shorter intervals: These give the OL more opportunities to influence the popula-
tion. (2) Larger populations: These make outliers with exceptional non-myopic performance
more likely, and OL makes them likely to survive and propagate.
The baseline (no meta-learming) algorithms all pass the unit tests: hidden incentives almost never revealed see blue curves in Fig. However, agents trained with meta-learning and large populations often fail the unit tests: see orange curves in top rows of Fig. 4
PBT
PBT interval =
PBT interval
PBT interval 100
#a
#agents=100
aggents
1.00
0.75
0.50
0.25
3.00
3.00
400
100
1000
200
400
600
820
1002
200
400
100
802
1000
time-lep
(A) Myopic RL Unit Test. OL=PBT.
#agents=10
#agents-100
#apents-1000
(looperate)
0.75
0.00
1.00
ploooaraae
0.73
0.25
0.00
0
100
400
600
800
1000
200
400
800
800
1000
2CD
4001
600
800
1000
timesttp
tmm-step
time stme-stp
(B) Myopic RL Unit Test. OI=REINFORCE
Figure 4: Average level of non myo-mc cooperate behavior observed in the RL unit test, with two meta-Icarning algorithms (A) PBT and (B) REINFORCE, Lower is better, since the goal is for (non- myopic) incentives for ADS to remain hidden. Despite the inner loop being fully myopic (7 0), outer-loop (OL) optimizers reveal incentives for ADS (top rows). Context swapping effectively hides this incentive, reducing ADS (bottom rows).
Furthermore, we verify that context swapping significantly mitigates the effect of HI-ADS in both unit tests, decreasing undesirable behaviour to bear-baseline levels see bottom rows of Fig. 4 This effect can be explained as follows: Because context swapping transfers the benefits of one learner's action to the next learner to inhabit that environment, it increases the second learner's fitness, and thereby reduces the relative fitness (as evaluated by PBT's EXPLOIT step) of the non- myopic cooperate behaviour. We observe some interesting exceptions with the combination of small populations and short PBT intervals: Although context swapping still significantly decreases the effect of HI-ADS, non- myooic cooperate behaviour observed as much as 20% of the time (for #leamnes==00, T 1; bottom-left plot).

---

Under review as a conference paper at ICLR 2022
We also observe that PBT reveals incentives for ADS even when T = 1, where the explanation that PBT operates on a longer time horizon than the inner loop does not apply. We provide a detailed explanation for how this might happen in Appendix C.I.3 but in summary, we hypothesize that there are at least 2 mechanisms by which PBT is revealing incentives for ADS: (1) optimizing over longer time-scale, and (2) picking up on the correlation between an agent's current policy and the underlying state. Mechanism (2) can be explained informally as reasoning as: "If I'm cooperating, then was probably cooperating on the last time-step as well, so my reward should be higher". As support for these hypotheses, we run control experiments identifying two algorithms (each sharing only one of these properties) that can fail the unit test. Context swapping remains effective in both.
(I) Optimizing over longer time-scale: replacing PBT with REINFORCE as an outer-loop op- timizer. The outer-loop optimizes the parameters to maximize the summed reward of the last T time-steps. As with PBT, we observe non-myopic behavior, but now only when T > 1. This sup- ports our hypothesis that revealing incentives for ADS is due not to PBT particular, but rather to the introduction of sufficiently powerful meta-learning, See Fig. 4 B2.
(2) Exploiting correlation: Q-Jearning with 7 = 0 an e = 0. 0.1-greedy behavior policy and no meta-leurning. If either state was equally likely, the Q-values would be the average of the values in each column in Table 1, so the estimated Q(defect) would be larger. But the e-greedy policy correlates the previous action (i.c. the current state) and current action (so long the policy did not just change), so the top-left and bottom-right entries carry more weight in the estimates, sometimes causing ((defect x (ccopeeratt rate) and persistent nonmyopic behavior. See Fig. 5 results, and Appendix CCI.44 for experimental details. A similar effect is observed in offline RL, even though the learner cannot influence its data distribution, see Appendix C.1.5 for details.
0.0
0.0
Picooperate) Q-values
0.00
0.0
0.2
-0.25
-05
-04
0.4
-0.50
06
1.0
10
1.0
1.0
1.00
0.9
0.9
0.95
0.90
02
0.8
0.0
2000
2000
0
2000
0
2000
2000
00
time-step
time-step
time-step
time stee-step
time-step
Q(defect)
Occooperate)
#cooperate/lime-step
Figure 5: Left: Offline Q-learning can reveal incentives for ADS when pooling data from different
policies. Yellow regions represent policy pairs (0,, 02) for which Q(C') > QD)) in the Myopic RL unit test, resulting in non non-myopic behavior. Right: Even online, Q-learning fails the unit test for some random seeds; empirical p (cooperate) stays around 80-90% in 3 5 experiments (bottom row). Each column represents an independent experiment. Q-values for the cooperate and de fect actions stay tightly coupled in the failure cases (col. 1,2,5), while in the cases passing the unit test (col. 3,4) the Q-value of cooperate decreases over time.
5.2 INCENTIVES FOR ADS IN CONTENT RECOMMENDATION
We now present a toy environment for modeling content recommendation of news articles, which includes the potential for ADS by incorporating the mechanisms mentioned in Sec. 2.2 discussed as contributing factors to the problems of fake news and filter bubbles. Specifically, the environment assumes that presenting an article to a user can influence (1) their interest in similar articles, and (2) their propensity to use the recommendation service. These correspond to modeling auto-induced concept shift of users, and auto-induced covariate shift of the user base, respectively (see Sec. 2.2)
This environment includes the following components, which change over (discrete) time: x', Article type: y², User interests: W' (propensity for users of each type to click on articles of each type), and User loyalty: g' (propensity for users of each type to use the platform). At each time step t, a user is sampled from a categorical distribution, based on the loyalty of the different user types. The recommendation system (a classifier) selects which type of article to present in the top position, and finally the user 'clicks' an article y², according to their interests. User loyalty for user type r² undergoes covariate shift: in accordance with the self-selection effect, g¹ increases
User type:

---

Under review as a conference paper at ICLR 2022
decreases proportionally to that user type's interest in the top article. The interests of user type x' (represented by a column of W undergoing concept shift; in accordance with the illusory truth effect, interest in the topic of the top article chosen by the recommender system always increases.
Formally, this environment is similar to a POMDP R. i.c. a POMDP with no reward function, also known world model (Armstrong & O'Rourke 2017 Madfield al. (017); the differ- ence is that the learner observes the input (o before acting and only observes the target (o (opss)
after acting. The states 8, observations o, and actions a are computed follows:
s' (g' W x',,"""" opre: a', of (x v'
For further details on this environment, including the state transition function, see Appendix C.2.1
5.2.1 CONTENT RECOMMENDATION EXPERIMENTAL RESULTS AND DISCUSSION
We find that PBT yields significant improvements in training time and accuracy, but also greater distributional shift (Fig. (6) User base and user interests both change faster with PBT, and user interests change more overall. We observe that the distributions over user types typically saturate (to single user type) after few hundred time- (Fig D and Fig- 6 Right). We long enough to reach such states, to demonstrate that the increase in ADS from PBT is not transitory. The environment has number of free parameters, and our results are qualitatively consistent so long as (I) the initial user distribution is approximately uniform, and (2) the covariate shift rate (a1) ( is faster than the concept shift rate (a:2). See Appendix C.I for details.
We measure concept shift (change in P(y)x) as the cosine distance between each user types' initial and current interest vectors. And we measure covariate shift (change in P(x)) as the KL-divergence between the current and initial user distributions, parametrized by g! and g', respectively. Our recommender system is a I-layer MI.P trained with SGD-momentum. Actions are sampled from the MLP's predictive distribution. For PBT, we use T = 10 and 20 agents, and use accuracy to evaluate performance. We run 20 trials, and match random seeds for trials with and without PBT. See Appendix C.2 for full experimental details.
prodiction
PLYP) intarosts)
Change PUX) Peeeeeee
cOStee gisternce Irom original interets
2.0
0.05
C.drr from orgina dsseibution
2.0
0.03
1.0
0.7
0.03
1200
PMT
- no PLT
Figure 6: Content recommendation experiments. Left: using Population Based Training (PBT) increases accuracy of predications faster, leads to a faster and larger drift in users' interests, P(y]x), (Center); as well as the distribution of users, P(x), (Right). Shading shows std error over 20 runs.
6 RELATED WORK
ADS in practice: We introduce the term ADS, but we are far from the first to study it. Caruana al. (2015) provide an example of asthmatic patients having lower predicted risk of pneumonia. Treating asthmatics with pneumonia less aggressively on this basis would be an example of harmful ADS; the reason they had lower pneumonia risk was because they had received more aggressive care already. SSchulam & Sarial (011)) note that such predictive models are commonly used to inform decision- makiin, and propose modeling counterfactuals (e.g. "how would this patient fare with less aggressive treatment") to avoid such self-refuting predictions. While their goal is to make accurate predictions in the presence of ADS, our goal is to identify and manage incentives for ADS. Goodfellow (2019) argues that adversarial defenses that do not account for ADS critically flawed.
8

---

Under review as a conference paper at ICLR 2022
Non-i..d bandits:
Contextual bandits (Wang et al. 2005 Langford & Zhang 2008) are frequently discussed as an approach to content recommendation (Li et al. 2010). While bandit algorithms
typically make the i..d. assumption, counter-cxamplee
exist (Gheshlaghi Azar al. 2014 Shah
et al. 2018); most famously, adversarial bandits (Auer et al. 1995). Closest to our work is Shah et al. (2018), who consider covariate shift caused by multi-armed bandits. Our task in Sec. [5.2 similar
to their problem statement, but more general in that we include user features, thus disentangling covariate shift and concept shift. Our motivation is also different: Shah et al. (2018) seek exploit
ADS, whereas we aim to avoid hidden incentives for ADS.
Safety and incentives: Emergent incentives to influence the world are the heart of many con-
cerns about the safety of advanced Al systems (Omohundro) [2008] Bostrom 2014). Understanding
and managing the incentives of learners is focus of Armstrong & O' Rourke (2117) Everitti (2018);
Everitt et al. (2019); Cohen et al.] (2019). While Everitt et al. (2019) focus on identifying which incentives are present, we note that incentives may be present and yet not be revealed or pursued for example, in supervised learning, there is an incentive to over-fit fit the test set, but hiding the test set from the learner hides this incentive. While Carey et al. 2020); Everitt et al. (2019) Armstrong & O'Rourke (217)) discuss methods of removing problematic incentives, note in practice in- centives are often hidden rather than removed. Our work addresses the efficacy of this approach of hiding incentives and ways in which it can fail.
Incentives and meta-learning: We believe our work is the first to consider the problem of hid- ing/revealing incentives for ADS, and the relation to meta-learning A few previous works have some relevance or resemblance. |Rabinowitz] (2019) documents qualitative differences in learning behavior when meta-learning is applied. Mac et al. (2019) and Lorraine & Duvenaud (2018) view meta-learning as a bilevel optimization problem, with the inner loop playing a best-response to the outer loop. In our work, the inner loop is unable achieve such best-response behavior; the outer loop is too powerful (see Fig. 4 Finally, /Sutonn et al. (2007) note that meta-learning can change learning behavior and improve performance by preventing convergence of the inner loop.
source of poor behavior in real world settings. They focus on differences in training vs. deployment performance, similarly to [lyaa et al.) 2019) Koch et al. (02I). We go beyond this by showing how changing which incentives are revealed can lead to fundamentally different solutions with different training performance.
Underspecification: D' Ammouur et al. (2000) discuss underspecification
7 DISCUSSION AND CONCLUSION
We have identified the phenomenon of auto-induced distributional shift (ADS), and the problems that can arise when previously hidden incentives for learners to induce distributional shift are revealed. Our experiments demonstrate that using meta-learning can reveal incentives for ADS, leading learn- ers to use ADS as a means of increasing performance.
Our work highlights the interdisciplinary nature of issues with real-world deployment of ML systems we show how revealing incentives for ADS could play a role in important technosocial issues like filter bubbles and the propagation of fake news. There
number of potential implications for our work: (1) When ADS are a concern, our methodology and environments can be used to help diagnose whether and to what extent the final performancc/hehavior of a learner is due ADS and/or incentives for ADS, i.e. to quantify their influence on that learner. (2) Comparing this
quantitative analysis for different algorithms could help us understand which features of algorithms affect their propensity to reveal incentives for ADS, and aid in the development of safer and more robust algorithms.
(3) Characterizing and identifying incentives for ADS in these tests is first step to analyzing and mitigating other (problematic) incentives, as well as to developing theoretical understanding of incentives.
Our work emphasizes that the choice of machine learning algorithm plays an important role in specification, independently of the choice of performance metric. A learner can use ADS to increase performance according to the intended performance metric, and yet still behave in an undesirable way, if we did not intend the learner to improve performance by that method. In other words, performance metrics are incomplete specifications: they only specify our goals or ends, while our choice of learning algorithm plays a role in specifying the means by which we intend an learner to achieve those ends. With increasing deployment of ML algorithms in daily life, we believe that (1) understanding incentives and (2) specifying desired/allowed means of improving performance are important avenues of future work ensure fair, robust, and safe outcomes.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Hunt Allcott and Matthew Gentzkow. Social media and fake news in the 2016 election. Journal of Economic Perspectives, 31 1-,6,, May 2017.
Michelle A. Amazeen and Bartosz W. Wojdynski. Reducing native advertising deception: Revis- iting the antecedents and consequences of persuasion knowledge in digital news contexts. Mass Communication and Society, 0(0):1-26, 2018.
arXiv preprint Stuart Armstrong and Xavier O'Rorke. Good and safe uses oracles. arXiv:1711.05541 2017.
Stuart Armstrong and Xavier O'Rourke Indifference methods for managing agent rewards. Tech- nical report, Future of Humanity Institute, 2017.
K. J. Åström. Optimal control of Markov Processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10:174-205, January 1965.
P. Auer, N. Cesa-Bianchi Y. Freund, and R. E. Schapire. Gambling in rigged casino: The adver- sarial multi-armed bandit problem. In Foundations of Computer Science, 1995.
Eytan Bakshy, Solomon Messing, and Lada A. Adamic. Exposure to ideologically diverse news and opinion on Facebook. Science, 34866239);1133-11322 2015. ISSN 0036-8075, doi: 10. 00.1126/science.aan aa URL
wwwp::/wwww.ciiccccce...... http://mmmmmmrrercooooottet//// content/348/
6239/1130
Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Inc., New York, NY, USA, 1st edition, 2014.
Ryan Carey, Eric Langlois, Tom Everitt, and Shane Legg. The incentives that shape behaviour. arXiv preprint arXiv:2001.07112 2020.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intel- ligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In International Conference Knowledge Discovery and Data Mining, pp. 1721-1730, 2015.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2017.
Michael K. Cohen, Elliot Catt, and Marcus Hutter. A strongly asymptotically optimal agent in general environments. Proceedingy of the Twenty-Fighth International Joint Conference Ar- tificial Intelligence, Aug 2019, doi: 10.24963/jccai22119/00 URL http:/ /xx doi. org// 10. 10.24963/1jca1.200 /ttp:////aaa2.2.//...
Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen. Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffmman, Farhad Hormozdi- ari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingan, Mario Lucic, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymy- rov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Underspecification presents challenges for credibility in modern machine learning, 2020.
Dominic DiFranzo and Kristine Gloria-Garde Filter bubbles and fake news. XRDS. 23(3):32-33 35, April 2017. ISSN 1528-4972. doi: 10. 10.1145/3055153, URL http::/ddat acm. org/10. 1145/3055153
Mostafa M. El-Bermawy. Your echo chamber destroying democracy, 2016. URL https:// wi red. com /tt.////11////// /tt memmoeeerry
Tom Everitt. Towards Safe Artificial General Intelligence. PhD thesis, Australian National Univer-
sity, 2018
10

---

Under review as a conference paper at ICLR 2022
Tom Everitt and Marcus Hutter. The alignment problem for bayesian history-based reinforcement learners. 2018.
Tom Everitt and Marcus Hutter. Reward tampering problems and solutions in reinforcement learn- ing: A causal influence diagram perspective. arXiv preprint arXiv:1908.047:4 2019.
Tom Everitt, Pedro A. Ortega, Elizabeth Barnes, and Shane Legg. Understanding agent incentives using causal influence diagrams, part i: Single action settings, 2019.
Lisa K. Fazio, Nadia M. Brashier, B. Keith Payne, and Elizabeth J. Marsh. Knowledge does protect against illusory truth. Journal of Experimental Psychology: General, 144(5):993-1002, 10 2015. ISSN 0096-3445. doi: 10. 10.103/xge00000088
Seth Flaxman and Sharad Goel. Filter bubbles, echo chambers, and online news consumption. Public Opinion Quarterly, 2015.
Mohammad Gheshlaghi Azar, Alessandro Lazaric, and Emma Brunskill. Online stochastic opti- mization under correlated bandit feedback. ArXiv preprint, 2014.
lan J. Goodfellow. A research agenda: Dynamic models to defend against correlated attacks. ArXiv preprint, 2019.
Jacob Groshek and Karolina Koc-Michalske Helping populism win? Social media use, filter bub- bles, and support for populist presidential candidates in the 2016 us election campaign. Informa- fion, Communication & Sociery,
2017. doi: 10. 10.1000/13691 /0. 1wt.:1000011110
URL http://oooo.rrr..o..000///111111
2017. 03773332334
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. In Neural Information Processing Systems, 2017.
Andrew lyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features, 2019.
M. Jaderberg, V. Dalibard, S. Osindero, W. M. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, Dunning, K. Simonyan, C. Fernando, and K. Kavukcuoglu. Population Based Train- ing of Neural Networks. ArXiv preprint, 2017.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in par- tially observable stochastic domains. Artificial Intelligence, 101(1-2):99-134, May 1998. ISSN 0004-3702.
Varol Kayhan. Contirmation bias: Roles of search engines and search contexts. In International Conference on Information Systems, 2015.
Jack Koch, Lauro Langosco, Jacob Pfau, James Le, and Lee Sharkey, Objective robustness in deep reinforcement learning, 2021.
Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ra- mana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of ai ingenuity, 2020. URL /nttps:://eepnmt wttpa//dd..oo/moo//aa.iiioo Speci f caaccaiionn gaming the-flip welllp-siiee offidd-fff-foo - ingenuity
John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Neural Information Processing Systems, 2008
Ed. Lee Howell. Digital
wildfires
a hyperconnected world.
Lee URL
Howell (ed.), Global Risks
2013. World Economic Forum, 2013.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq,
Laurent Orseau, and Shane Legg. Al safety gridworlds. Technical report, DeepMind, 2017.

---

Under review as a conference paper at ICLR 2022
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. Technical report, DeepMind Safety Research, 2018.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems, 2020.
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In International Conference on World Wide Web, 2010.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypemnet- works. ArXiv preprint, 2018.
D. D. Luxton, J. D. June, and J. M. Fairall. Social media and suicide: A public health perspective. American journall of public health, 102(2):195-206 2012.
Matthew MacKay, Paul Vicol, Jonathan Lorraine, David Duvenaud, and Roger Grosse. Self- tuning networks: Bilevel optimization of hyperparameters using structured best-response func- tions. ArXiv preprint, 2019.
Merriam-Webster.
URL /ttps:///ww. merriam- weetnn-wbssee.... com words-waaaeooooo at ptaaythhe-eeeeeeeeeeeeeeeee -et--atoy----- news
The real story of fake news, 2017.
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Learning unsuper-
vised learning rules. In International Conference on Learning Representations, 2019.
Paul Mihailidis and Samantha Viotty. Spreadable spectacle in digital culture: Civic expression, fake news, and the role of media literacies in "post-fact" society. American Behavioural Scientist, 2017.
Jose G. Moreno-Torres, Troy Raeder, RocíO Alaiz-RodriGuez, Nitesh V. Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern Recognition, 45(1):521-530, January 2012.
Tien T. Nguyen, Pik-Mai Hui, F. Maxwell Harper, Loren Terveen, and Joseph A. Konstan. Exploring the filter bubble: The effect using recommender systems on content diversity. In Proceedings of the 23rd International Conference World Wide Web, WWW 14, pp. 677-686, New York, NY, USA, 2014. ACM. ISBN 978-1440-44452 doi: 10.1 041144226664862566000 URL
7/doi. acm. org
2568012
Safiya Umoja Noble. Algorithms of Oppression: How Search Engines Reinforce Racism. NYC Press, 2018.
Stephen M. Omohundro. The basic AI drives. In Conference Artificial General Intelligence, 2008.
Pedro A. Ortega, Vishal Maini, et al. Building safe artificial intelligence: specification, robustness, and assurance, 2018
Eli Pariser. The Filter Bubble: What the Internet Is Hiding from You. The Penguin Group. 2011.
Gordon Pennycook, Tyrone D Cannon, and David G. Rand. Prior exposure increases perceived accuracy of fake news. Journal of Experimental Psychology (forthcoming). 2019.
Erich Prisner. Game Theory Through Examples. Mathematical Association of America, 2014.
Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence. Dataset Shift in Machine Learning. The MIT Press, 2009.
Neil C. Rabinowitz. Meta-learners' learning dynamics are unlike learners'. ArXiv preprint, 2019.
2019.
David Robson. The myth of the online echo chamber, 2018. URL http://wwwwbbbcccc com/ htt.ee--o-nn--eecc-----aaabeee
Futuree/story/201860
12

---

Under review as a conference paper at ICLR 2022
Peter Schulam and Suchi Saria. Reliable decision support using counterfactual models. In Neural Information Processing Systems, 2017.
Virag Shah, Jose Blanchet, and Ramesh Johari. Bandit learning with positive externalities. Neural Information Processing Systems, 2018.
Chengcheng Shao, Giovanni Luca Ciampaglia, Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menezer. The spread of low-credibility content by social bots. Nature Communica- tions, (4777)), 2018.
Richard Sutton and Andrew G Barto. Introduction to Reinforcement Learning. MIT Press, 1998.
Richard S Sutton, Anna Koop, and David Silver. On the role of tracking in stationary environments. In International conference Machine learning, 2007.
Techopedia. Filter bubble, 2018. URL /ttp://////ww...coppaaaaaaaaa/aa
28556/fliter-ubbbll
Chih-Chun Wang, Sanjeev R Kulkarni, and H Vincent Poor. Bandit problems with side observations. IEEE Transactions on Automatic Control, 50(3):33--355 2005.
Wikipedia contributors. Confirmation bias Wikipedia, the free encyclopedia, 2018. URL
https: wiki www.wwwwwwww.......cc.......... org/ w/ www.ndeepi..co wtt.:pphptitteee.omm.. rmat blass o oldid-875026726 [Online; accessed 20-Jamuary-2019].
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinfrrmmeett learning. Machine Learning, pp. 229-256, 1992.
13

---

Under review as a conference paper at ICLR 2022
A CONTENT RECOMMENDATION IN THE WILD
Filter bubbles, the spread of fake news, and other techno-social issues are widely reported to be responsible for the rise of populism (Groshek & Koc-Michalska 2017), increase in racism and prejudice against immigrants and refugees (Noble 2018), increase in social isolation and suicide (Luxton et al., 2012). and, particularly with reference to the 2016 US elections, are decried as threatening the foundations of democracy (EI- (El-Bermawy 2016). Even in 2013, well before the 2016 American elections, World Economic Forum report identified these problems as global crisis
(Lee Howell| 2013).
We focus on two related issues in which content recommendation algorithms play a role: fake news and filter bubbles.
A.1 FAKE NEWS
Fake news (also called false news or junk news) is an extreme version of yellow journalism, pro- paganda, or clickbait, in which media that is ostensibly providing information focuses on being eye-catching or appealing, at the expense of the quality of research and exposition of factual infor- mation. Fake news is distinguished by being specifically and deliberately created spread false- hoods or misinformation (Merriam- Webster 2017, Mihailidis & (017).
Why does fake news spread? It may at first seem the solution simply to educate people about the truth, but research tells us the problem is more multifaceted and insidious, due to a combination of related biases and cognitive effects including confirmation bias (people are more likely to believe things that fit with their existing beliefs), priming (exposure to information unconsciously influences the processing of subsequent information, i.e. seeing something in a credible context makes things seem more credible) and the illusory truth effeet (i.e. people are more likely to believe something simply if they are told it is true).
Allcott & Gentzkow (017) track about 150 fake news stories during the 2016 US election, and find the average American adult saw 1-2 fake news stories, just over half believed the story was true, and likelihood of believing fake news increased with ideological segregation (polarization) of their social media. [Shao et al.] (2018) examine the role of social bots in spreading fake news by analyzing 14 million Twitter messages. They find that bots are far more likely than humans to spread misinformation, and that success of fake news story (in terms of human retweets) was heavily dependent on whether bots had shared the story.
Pennycook et al. 2019) examine the role of the illusory truth effect in fake news. They find that even a single exposure
news story makes people more likely to believe that it true, and repeat viewings increase this likelihood. They find that this is not true for extremely implausible statements (e.g. "the world a perfect cube"), but that "only a small degree of potential plausibility is sufficient for repetition to increase perceived accuracy" of the story. The situation is further complicated by peoples' inability to distinguish promoted content from real news - Amazzen & Wojdynski] (2018) find that fewer than 1/10 people were able to tell when content was an advertisement, even when it was explicitly labelled as such. Similarly, Fazio et al. (2015) find that repeated exposure to incorrect trivia make people more likely to believe it, even when they are later able to identify the trivia as incorrect.
A.2 FILTER BUBBLES
Filter bubbles, a term coined and popularized by Pariser (201T) are created by positive or negative feedback loops which encourage users groups of users towards increasing within-group similarity, while driving up between- groupp dissimilarity. The curation of this echo chamber called self- selection (pcople favours what Techopedia (2018) calls intellectual isolation. In the context of social and political opinions, this is often called the polarization effect (Wikipedia contributors 2018).
more likely to look for or select things that fit their cxisting preferences), and
Filter bubbles can be encouraged by algorithms in two main ways. The first is the most commonly described: simply by showing content that is similar to what user has already searched for, search or recommender systems create a positive feedback loop of ancresingll-simiil content (Pariser) 2011 Kayhan 2015). The second way is similar but opposite if the predictions of algorithm
14

---

Under review as a conference paper at ICLR 2022
good for certain group of people, but bad for others, the algorithm do better its metrics by driving hard-to-predien users away. Then new users to the site will either be turned off entirely, see artificially homogenous community of like-minded peers, phenomena Shah et al. (2018) call positive externalities.
In study of 50,000 US-based internet users, Flaxman & Goel (2013) find that two things increase with social media and search engine use: (1) exposure of an individual to opposing or different viewpoints, and (2) mean ideological distance between users. Many studies cite the first result as evidence of the benefits of internet and social media (Robson 2018 Bakshy et al. 2013), but the
correlation of exposure with ideological distances demonstrates that exposure is not enough, and might even be counterproductive.
Facebook's 's own study on filter bubbles results show that the impact of the news feed algorithm on filter bubble "size" (a measure of homogeneity of posts relative to a baseline) is almost as large as the impact of friend group composition (Bakshy al.) (013).. Kayhan (2015) specifically study the role of search engines in confirmation bias, and find that search context and the similarity of results search engine results both reinforce existing biases and increase the likelihood of future biased searches. Nguyen et al. (2014) similarly study the effect of recommender systems individual users' content diversity, and find that the set of options recommended narrows over time.
Filter bubbles create an ideal environment for the spread of fake news: they increase the likelihood of repeat viewings of similar content, and because of the illusory truth effect, that content is more likely to be believed and shared (Pennycook al. 2019 DiFranzo & Gloria-Garcial 2017 Pariser 2011]. We are not claiming that incentives for ADS are entirely or even mostly responsible for these problems, but do note that they can play a role that is worth addressing.
15

---

Under review as a conference paper at ICLR 2022
incentive-compaubble (3 = 0.5)
#agents=10
#agents=100
#agents=1000
1.00
Eurddwws AUs
Averbose)
0.75
0.50
0.25
0.00
1.00
Burddlww AUd
Piverbooe)
0.75
0.50
0.25
time-step
time-step
time-step
incentive-orthogonal (B 0.0)
#agents=10
#agents=100
#agents=1000
1.00
Gurddwww AUs oc
Pveebooe))
0.75
0.25
1.00
Averbooe))
0.75
Buiddews AUB
0.50
0.25
0.00
250
500
750
1000
250
500
750
1000
250
500
750
1000
time
step
incentive-opposed (3= -0.5)
#agents=10
#agents=100
#agents=1000
1.00
Gurddewee MUO 0U
Fiverbose)
0.75
0.50
0.25
0.00
Bunddeows AuD
Xveebose)
0.75
0.50
0.25
0.00
250
500
750
1000
250
500
750
1000
250
500 time-step
750
1000
time-step
time-step
no PBT
PBT interval =1
PBT interval
10
PBT interval 100
Figure 7: Average level of non-myopic (i.e. cooperate) behavior learned by agents in the unit test for incentives for ADS. Despite making the inner loop fully myopic (? 0), population-base training (PBT) can reveal incentives for ADS, leading agents to choose the coope rate action (top row). context swapping successfully prevents this (bottom row). Columns (from left to right) show results for populations of 10. 100, and 1000 learners. the legend, "interval" refers the interval (T) of PBT (see Sec. 2.2). Sufficiently large populations and short intervals are necessary for PBT induce nonmyopic behavior.
16

---

Under review as a conference paper at ICLR 2022
B ADS INCENTIVE UNIT TEST (SUPERVISED LEARNING)
This unit test consists of simple prediction problem. There are inputs, only underlying state 8 E {0, 1), and targets y E IR² with v1, 3/2 ~ (O, 8 NV),,((, 1), with corresponding predictions V1, f/2. Additionally, $141 0 iff 8/2 .5. We use Mean Squared Error the loss function, the optimal predictor is 1/1 1/2 = (). However, predicting 1/2 > .5 reduces the variance of y1, ie. reduces future loss.
The baseline/lL. predictor learns ÿ 1/2 as parameters using SGD with a learning rate 0.001. For experiments with meta-learning, PBT is the OL (with default settings, see Section 2.2), used to tune the learning rate, with negative loss on the final time-step of the interval as the performance measure for PBT.
no PBT
PBT interval
PBT interval
PST interval 100
#agenis=10
#agents=100
#agents=100
0.75
0.50
0.25
3.00
1.00
400
ND0
1000
200
400
600
B00
1002
200
-000
802
1000
Emne stop
time stop
Enne stop
Figure 8: Results on the Supervised Learning ADS unit test mirror those on the RL unit test. PBT produces larger values of f/2 sacrificing present performance for future performance (i.c. non- myopic exploitation ADS).
C EXTRA EXPERIMENTS AND REPRODUCIBILITY DETAILS
C.I ADS INCENTIVE UNIT TEST
C.I.1 FORMAL DEFINITION OF MYOPIC RL ADS UNIT TEST EN ENVIRONMENT
Formally, this environment is not 2x2 game (as the original prisoner's ' dilemma); it's partially
observable Markov Decision Process Astrrm 1965 Kaelbling et al. 1998):
Ot =
a--1, 0
at fdefect, cooperate] P(st, ax) = 8(ur)
R88 (t) = T(s cooperate) +
3 I(a = cooperate) - 1/2
where I is indicator function, and / -1/2 is parameter controlling the alignment of incen- tives (see Appendix 3.2 for an exploration of different 3 values.). The initial state is sampled as So U(defect, cooperate).
C.1.2 ALIGNMENT INCENTIVES EXPLORATION
This section presents an exploration of the parameter B, which controls the alignment of incentives in the Myopic RI. unit test (see Table 2).
To clarify the interpretation of experiments, we distinguish between environments in which myopic (defect) vs. nonmyopic (cooperate) incentives are opposed, orthogonal, or compatible. Note that in this unit test myopic behaviour (defection) is what we want to see.
Incentive-opposed: Optimal myopic behavior is incompatible with optimal nonmyopic be- havior (classic prisoner's dilemma; these experiments are in the main paper).
17

---

Under review as a conference paper at ICLR 2022
2. Incentive-orthoggnaal: Optimal myopic behavior may may not be optimal nonmyopic be- havior.
3. Incentive-compatibl:: Optimal myopic behavior is necessarily also optimal nonmyopic be- havior.
We focused on incentive-opposed environment (3 = -1/2) in the main paper in order to demon- strate that incentives for ADS be powerful enough to change the behavior of the system in an undesirable way. Here we also explore incentive-compatible and incentive-orthogome environments because they provide useful baselines, helping distinguish systematic bias towards nonmyopic behavior from other reasons (such as randomness or optimization issues) for behavior that does not follow myopically optimal policy.
C.1.3 WORKING THROUGH DETAILED EXAMPLE FOR PBT WITH T =1
To help provide intuition on how (mechanistically) PBT could lead to persistent levels of coopera- tion, we walk through a simple example (with no inner loop). Consider PBT with T and a pop- ulation of 5 deterministic agents A A1.... Aş playing cooperate and receiving reward of r(A,) = 0. Now suppose A₁ suddenly switches to play defect. Then r(Ai) = 1/2 on the next time-step (while the other agents' reward is still 0), and PBT's EXPLOIT step will copy Ai (without loss of gen- erality to A2). On the following time-step, r(A2) = 1/2, and r(Ar) = -1/2, so PBT will clone A to A1, and the cycle repeats. Similar reasoning applies for larger populations, and T > 1.
Table 2: 3 controls the extent to which myopic and nonmyopic incentives are aligned.
Environment
Cooperating
0
incentive-opposed
yields less reward on the current time-step (myopically detrimental)
incentive-othhognnal
does not affect the current reward (myopically indifferent) yields more reward on the current time-step (myopically beneficial)
0
incentive-compatibe
C.1.4 Q-LEARNING EXPERIMENT DETAILS
We show that, under certain conditions, Q-learning can learn (primarily) cooperate, and thus fails the Myopic RL unit test. We estimate Q-values using the sample-average method, which is guar- anteed to converge in the fully observed, tabular case (Sutton Barto 1998). The agent follows the e-greedy policy with 0.1. In order to achieve this result, we additionally start the agent off with one synthetic memory where both state and action are defect and therefor R(defect) = -5, and we hard-code the starting state to be cooperate (which normally only happens 50% of the time). Without this kind of an initialization, the agent always learns to defect. However, un- der these conditions, we find that 10/30 agents learned to play cooperate most of the time, with ((cooperate) and Q(defoct) both hovering around -0.07, while others learn to always defect, with Q (coopperate) -0.92 and Q(defect) -0.45. context swapping, however, prevents majority- cooperate behavior from ever emerging, see Figure T1
C.1.5 OFFLINE Q-LEARNING CAN REVEAL INCENTIVES FOR ADS
In practice, RL agents are often trained offiine (Levine al. 2020). Incentives for ADS can still be revealed in offline RL, even though the learner cannot influence its data distribution. In particular, while QD) > Q(C)) for data from single policy, this does not always hold when pooling data from different policies, see Figure 5 Intuitively, pooling data from policies is similar to collecting data from an e-greedy policy trained online (as in Figure 5) This sort of data and approach in very common in real world applications, including content recommendation. and more generally, "A/B testing", where 2 groups of users are assigned to different policies, in order to compare the policies' performance.
18

---

Under review as a conference paper at ICLR 2022
C.1.6 Q-LERRNING: FURTHER RESULTS
To give more representative picture of how often Qllaaning fails the unit test, run a larger set of experiments with Q-learning, results are in Figure [O] It's possible that the failure of Q-learning persistent, since we have proved otherwise, but we did run much longer experiments and still observe persistent failure, see Figure
0.0
0.0
P(cooperate) Q-values
0.0
-05
-0.5
0.5
1.0
1.0
1.0
0.5
0.0
50000
50000
50000
50000
50000
time-step time-step
time-step
time-step
time-step
O(defect)
O(cooperate)
#cooperate/time-step
Figure 9: The same experiments as Figures illustrate the persistence of non-myopic behavior.
10
run for 50,000 time-steps instead of 3000, to
19

---

Under review as a conference paper at ICLR 2022
P(cooperate) Q-values
0.0
0.0
0.0
-0.5
-0.5
-0.5
0.5
0.5
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
2500
2500
2500
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
O(cooperate)
#cooperate/time-step
0.0
0.0
0.0
0.0
Q-values
0.5
005
-05
0.5
P(cooperate)
1.0
1.0
1.0
1.0
1.0
0.5
0.5
0.9
0.9
2500
time-step
time-step Q(cooperate)
time-step
time-step
time-step
O(defect)
#cooperate/time-step
0.0
0.0
P(cooperate) Q-values
0.0
-0.5
0.5
0.5
-1
-05
0.5
1.0
0.0
2500
2500
0
2500
2500
0
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Olcooperate)
#cooperate/time-step
0.0
0.0
P(cooperate) Q-values
0.0
0.0
-0.5
0..
-1
0.5
-055
1.0
1.0
1.0
1.0
1.0
0.5
0.9
0.9
0.8
0.0
2500
2500
0
2500
2500
2500
time-step
time-step
time-step
time-step
time-step
Qddffect
O(cooperate)
#cooperate/time-step
0.0
0.0
Q-values
0.0
0.0
0.0
-0.5
-0.5
-0.5
0.5
1.0
Picooperate)
1.0
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.5
0.0
0.0
0.0
2500
2500
2500
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Q(cooperate)
#cooperate/time-step
Figure 10: More independent experiments with Q-learning, exactly following Figure fails the unit test in total of 10/30 experiments (including those from Figure 5).
Q-learning
20

---

Under review as a conference paper at ICLR 2022
0.5
0.0
0.0
Q-values
0.0
0.5
-05
0.5
-0.
-1
Picooperatel
1.0
1.0
1.0
1.0
1.0
0.0
0.0
2500
2500
2500
2500
0
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Q(cooperate)
#cooperate/time-step
0.0
0.0
Q-values
0.0
0.0
0.0
0.5
0.5
-0.5
-0.5
0.5
Picooperate)
1.0
1.0
1.0
1.0
1.0
0.5
0.5
0.5
2500
2500
2500
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Qicooperate)
#cooperate/time-step
0.0
0.0
0.0
Q-values
0.0
-.5
0.5
0.5
0.5
-1
P(cooperate)
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
2500
2500
2500
2500
2500
time-step
time-step
time-step
time-step
time-step
O(defect)
O(cooperate)
#cooperate/time-step
0.0
0.5
Q-values
0.0
-0.5
-05
-05
0.5
-1
P(cooperate)
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
0.0
2500
2500
2500
2500
2500
time-step
time-step
time-step
time-step
time-step
Q(defect)
Q(cooperate)
#cooperate/time-step
0.0
Q-values
-0.
-1
P(cooperate)
1.0
1
1.0
1
0.5
0.5
2500
2500 time-step
2500
2500
0
2500
time-step
time-step
time-step
time-step
Q(defect)
Occooperate)
#copperate/time-step
Figure 11: More independent experiments with Q-learning, exactly following Figure except also using context swapping. This leads to 100% success rate on the unit test.
21

---

Under review as a conference paper at ICLR 2022
C.2 CONTENT RECOMMENDATION
C.2.1 ENVIRONMENT DETAILS
The evironment has the following components:
1. User type, z²: categorical variable representing different types of users. The content rec- ommender conditions its predictions the type of the current user.
2. User loyalty, g": the propensity for users of each type to use the platform. User æ' is sampled from a categorical distribution with parameters given by softmax(g").
3. Article type, y²: categorical variable (one-hot encoding) representing the type of article selected by the user.
4. User interests, W matrix whose entries W 2,3 represent the average interest user of type have in articles of type y.
At each time step i, a user x' is sampled from a categorical distribution (based on the loyalty of the different user types), then the recommendation system selects which type of article to present in the top position, and finally, the user selects an article. The goal of the recommendation system is to predict the likelihood that the would click on each of the available articles, in order select the one which is most interesting to the user.
User loyalty for of then changes in accordance with the self- effect, increasing decreasing proportionally to their interest in the top article. The interests of user type x' (represented by column of W') also change; in accordance with the illusory truth effect, their interest in the topic of the top article (as chosen by the recommender system) always increases. Overall, this environment an extremely crude representation of reality, but allows us to incorporate both the effects of self-selection (via covariate shift), and the illusory truth effect (via concept shift).
Formally, this environment similar to POMDP R, i.e. a POMDP with no reward function, also known as world model (Armstrong & O' O'Rourke| 2017, Madfield-Menel et al. 2017); the difference is that the learner observes the input before acting and only observes the target after acting. The states, observations, and actions given below.
s = (g", W xt ,29) opee a', Opest = (x (x',0',0 v.y
The state transition function is defined by:
g. g + aa W z1.0'
W W = g¹
w 1412 x'
= Wt 0t +2;
IW 1 a²
x 1+1 softmax(g 1+1)
y softmax w
Where j' is the top article as chosen by the recommender, and ai,, 02 represent the rate of covariate and concept shift (respectively). The update for W merely increases the interest of user type x! in article type ğ', then normalizes the interests for that user type.
C.2.2 REPRODUCIBILITY DETAILS
For these experiments, the recommendation system is a ReLU-MLP with I hidden layer of 100 units, trained via supervised learning with SGD (learning rate 0.01) to predict which article user will select. Actions are sampled from the MLP's predictive distribution. We apply PBT without any hyperparameter selection (this amounts to just doing the EXPLOIT step), and an interval of 10, selecting on accuracy. We use population of 20 learners (whether applying PBT not), and match random seeds for the trials with and without PBT. We initialize g' and W to be the same across the 20 copies of the environment (i.e. the learners start with the same user population), but these values diverge throughout learning. For the environment, we set the number of user and article types both to 10. Initial user loyalties are randomly sampled from (0, 0.03), an 0.03, and 02 0.003.
22

---

Under review as a conference paper at ICLR 2022
C.2.3 CONTEXT SWAPPING IN CONTENT RECOMMENDATION
We believe context swapping is not appropriate for the content recommendation environment, since when the environments diverge, optimal behavior may differ across environments. Nevertheless, we ran experiments with it for completeness. The main effect appears be to hamper learning when PBT is not used, see Figure [2] Notably, it does not appear to significantly influence the rate or extent of ADS when combined with PBT.
C.2.4 EXPLORATION OF ENVIRONMENT PARAMETERS
In Figure 13 we examine the effect of the rate-of-change parameters (a1, Q2) of the content rec- ommendation environment on the results provided in the paper. As noted there, our results are qualitatively consistent long (I) the initial user distribution approximately uniform, and (2) the covariate shift rate (01) is faster than the concept shift rate (az). These distributions are updated by different mechanisms, and are not directly comparable. Concept shift changes the task more radically, requiring learner to change its predictions, rather than just become accurate on wider range of inputs. We conjecture that changes P(yla) must therefore kept smooth enough for the outer loop have pressure to capitalize on ADS.
Accuracy of click prediction
Change in P(PP) [user] Itaracts)
[
Change in PC3) (user base)
codtre dittrrce tro onighee intereets
12.03
dv. from criginal dittrtte buEoo
0.8
3.5
0.8
2.03
3.0
0.01
0.5
0000
0
400
800
1200
1800
2000
400
800
1200
1600
2000
400
000
1200
1600
2000
timestap
Immestap
snapping
PBT, swapping
Change in P(X) (user base)
cosime distance from original interests interests
Change in P(Y]X) (user interests)
KL div. from original oniginal distribution
0.05
2.305
2.300
2.295
0.02
2.290
0.01
2.285
0.00
0.6
0.8
1.0
0.00
0.25
0.50
0.75
1.00
accuracy
accuracy
PBT
no PBT
PBT, env. swapping
PBT, env. swapping
Figure 12: Context swapping doesn't have the desired effect in the content recommendation envi- ronment.
23

---

Under review as a conference paper at ICLR 2022
01 = 0.01, 02 = 0.001
01 = 0.1, 02
0.001
Accuracy prediction
Accuracy
23
03-
1200
2000
2003
-20
100
1200
28C0
2003
timestap
timeslap
Change POYJI (user intarsnss)
Cherge in P(Y2)) (uset interwsls)
0.03
0.03
0.02
0.00
0.00
300
1200
3600
2003
400
500
1200
1000
2000
time step
(im step
Change In PIX) (user tiaso)
Change PIK) (user besc)
2-
800
1200
no PS1
+
PM1
n1 P5
O1] 0.01, 02 0.01
Q1 = 0.1, a2 0.01
prediction
preciction
23
3.-
0.6
400
000
1000
3800
2003
400
000
1200
1600
2000
timestep
sme-llap
Change F(IJX (user interasts]
Change P(IJI) (user intorests)
0.30
0.075
0.000
0,025
tinesste
3me-step
Change
Change
1-
1200
2800
2003
400
1200
1800
2000
PST
no
Q1 0.01, 02 0.1
[1 0.1, 02 0.1
Accuraty clice prediction
Accuraty click prediction
1.0
1.0
0.0
0.0
500 stop
step
in PIY
in P(YII (user interests)
0.6
0.2
0.05
400
300
1200
1600
2003
-00
500
1200
3000
2003
Chargge
Change
0
400
200
2000
1600
2003
400
300
1200
18C0
2003
Figure 13: Content recommendation results for different values of 01, Q2
24


Reviews:
[{"review_id": "yMafck2Ae9_", "reviewer": "Reviewer_vVp6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I like the problem studied in the paper but I have two concerns about related work and the role of these unit tests for informing you about how to design algorithms.\n\n**Related work**\n\nI take some issue with the authors reintroducing an existing conceptual idea under a different name. The realization that models can induce distribution shifts is not novel and it has for example previously been conceptualized in the context of risk minimization at a similar level of abstraction under the term *performative shift*. \n\nLet me elaborate on this: I am not sure the authors are aware of the very related literature on performative prediction. Perdomo et al [ICML’20] define performative effects as a shift in distribution induced by the predictive model you deploy. Then, performative prediction extends classical supervised learning and formalized as a risk minimization problem over a model dependent distribution. The goal is to minimize the loss over the distribution that will surface from deploying the model. In this context you can think of the loss as measuring the accuracy of your model, but you could also think of it as a general utility function on the induced distribution.   \n\nI don’t understand how the conceptual idea of performative distribution shifts is different of what you newly introduce as “auto-induced distribution shift”. Introducing a new term for something that already exists is not a good practice. At least you should discuss how it is different and why it needs a new term, or otherwise simply adopt an existing term. \n\nFollow-up works of Perdomo et al have studied both, algorithms that are performatively-aware in that they incorporate the shift into their updates, and naïve retraining strategies that are oblivious to the shifts and myopically optimize on the current distribution. In line with your findings, it has been realized that this leads to fundamentally different solution concepts. I really think connecting more to this line of work would be very interesting and beneficial for both communities. I will mention a couple of references in the end.\n\n**The role of the unit tests**\n\n think the focus of this work on the incentives and strategies that the learner pursues to optimize its utility is very interesting. However I am not fully understanding what you hope the general conclusions from the outcome of your unit tests would be.\n\nThe simple unit-tests are designed to analyze an algorithm’s behavior in the presence of undesirable incentives to shift the distribution. These examples are simple enough so that algorithm behavior is easy to interpret on this example and allows for a comparison between algorithms. It is interesting, although not that surprising that the behavior differs across algorithms. \n\nThe term unit tests implies that you want good algorithms to pass it. So this would mean you blame them for finding this alternate solution. Is that what your intention is? Do you think you can avoid this behavior without hurting the performance of the algorithm along other dimensions? As you mentioned in the introduction it does not seem to be a problem of the algorithm, but a problem specification issue. \n\nIn your examples the prediction task can be solved myopically and specification gaming requires taking observations of induced distributions into account. So what you will figure out with your test is whether your algorithm can learn from this feedback or not. Are you advocating for algorithms be 'bad' if they can do this? Would be nice if you could elaborate more on how you think these tests could help for certifying or improving your algorithms. \n\nI have a few additional questions I am curious about:\n-\tDo you think that in general specification gaming leads to larger (or worse) shifts than myopic algorithms? even if you optimize myopically you will inevitably cause shifts that will drive the distribution to a different state over time. \n-\tIt seems to me that meta-learning has more options to detect such actions, but it can only take advantage of this if the different configurations are exploring sufficiently. Is this correct?\n-\tYou mention that it is hard to specify an objective to anticipate such shifts. If you are aware of these shifts, would it be possible to use constraints or some sort of regularization to bias the solution towards a desirable induced distribution?\n-\tIt is not clear what Figure 4 (top row) is showing- what is $\\|y_2\\|$ referring to?\n\nSome references:\n\n- *Perdomo et al.* Performative prediction. 2020\n- *Miller et al.* Outside the Echo Chamber: Optimizing the Performative Risk. 2021\n- *Izzo et al.* How to learn when data reacts to your model: performative gradient descent. 2021\n- *Brown et al.* Performative Prediction in a Stateful World. 2020", "summary_of_the_paper": "The paper realizes that the ability of a model to induce a distribution shift can provide incentives to the learner that are not necessarily in line with the intention of the algorithm designer. It might find other means to reduce the risk through specification gaming rather than improving predictions. The authors propose two simple unit tests to detect such behavior and illustrate in these examples that meta-learning makes algorithms more susceptible to this potentially unanticipated behavior. As an approach to partially fix the issue they propose context-swapping such that the algorithm does not directly see the consequences of its actions. ", "summary_of_the_review": "I think the incentives perspective offered by this paper on model-induced distribution shifts is interesting. Designing simple tests enables the authors to provide some interesting insights about the relative performance of algorithms. However I am not sure what the algorithm designer can learn from the results of these unit tests. My second concern is the treatment of related work. I don’t think you should claim the definition of this phenomena as a novel contribution. \n\nI am willing to increase my score if the authors provide a satisfactory solution of how to address the conceptual overlap with existing work and they can convince me about the usefulness of these unit tests to guide algorithm design.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i9r0GZ87cAF", "reviewer": "Reviewer_nHgz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " - I could be wrong, but this paper to me is but an empirical extension to Performative Prediction (2020). Performative Prediction (2020) studies Repeated Risk Minimization, which to me is PBT (based on the author's description) but with only one learner; there is no parameter perturbing and replacing going on, only the retraining is being performed. As expected, there is a distribution shift when you do Repeated Risk Minimization. With PBT, we now have multiple learners doing the same thing. I don't find that PBT induces distribution shift surprising at all. The utility function of the data studied in this paper is more complicated than that in Performative Prediction (2020), but I don't see it matters for the main findings of this paper. \n\n - We, the machine learning engineers, are meta-learners. We tune parameters of models, deploy them in the field, and observe distribution shifts. The main message of this paper, in my opinion, did not need to be motivated by meta-learning or PBT at all; the message could have been made more powerful without it. PBT can be added in the experiments as a way to simulated a meta-learner, i.e. human. \n\n - For a paper that uses game theory terminology in its title, I find it hard to follow its game theory elements. The necessary notations are introduced very late, in Sec. 5.2. There is also an overuse of italic fonts, which have me wondering if those are concepts that the authors will define concretely using notations later in the article (which they aren't).\n\n - \"Revealed incentive\" sounds odd. In game theory, the agents' incentives are well defined: the classifier wants higher accuracy, and the data wants, say, better movie recommendations. The game is then played, and the equilibrium is then observed. One can argue that the classifier causing a distribution shift is an unwanted equilibrium of this game, but saying that the change of the data's equilibrium strategy in this sequential game is an \"incentive\" of the classifier being \"revealed\" is ... not standard.\n\n - Typo: Captions to Figure 2: state s, action s, reward s tuples.", "summary_of_the_paper": "This paper proposes to study Auto-induced Distribution Shift (ADS), the phenomenon that models can create a feedback-loop: the prediction of a model influence user behaviors when it is deployed and retrained iteratively, which, in turn, affects the accuracy measure of the model. The paper empirically shows that a meta-learning algorithm called PBT causes a distribution shift instead of maximizing accuracy. ", "summary_of_the_review": "I believe this paper is an empirical, meta-learning extension to Performative Prediction (2020). I believe the main findings of the paper are not attributed to meta-learning, but to the idea of repeated retraining for online learning, which is well explored in Performative Prediction (2020). In addition, this paper does not cite Performative Prediction (2020), and the writing (especially the use of notations and terminologies) needs work. Therefore, I incline to recommend rejection. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "byGcCYb2GY0", "reviewer": "Reviewer_yBdS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- the topic of the paper is important and interesting. In today's world where ML algorithms are deployed on humans, it is a stretch to assume i.i.d. or static user data.\n- there is novelty in the paper in that the goal is not necessarily to deal with or compensate for ADS, like in much related work, but rather to try to detect incentives for the learner to induce ADS.\n\nWeaknesses:\n- this seems very, very related to the idea of strategic behavior in machine learning, and of \"Performative Prediction\" (Juan C. Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, Moritz Hardt). I could not find this line of work in the citations of the paper, and I think it may need to be there. Currently, the paper ignores an important line of work that considers exactly the same effects (though maybe in a slightly different setting(.\n- I do not think the main insights of the paper are super surprising or new. I don't think there is really anything novel in observing that PBT reveals incentives better -- obviously PBT will do \"well\" simply through the fact that it heavily incentivizes exploration. It is also not surprising that some ML algorithms can learn to reveal this incentive even if they look at myopic utilities: for example, imagine a 2-player game in which one player (so here it could the learner in the previous time step) does best response and the other one (the learner in the current time step) plays a no-regret algorithm (but the loss function and the update rule at a given round only depends on the utility in this round, and not the future utilities -- for example multiplicative weights); then it is known that such an algorithm will converge to an equilibrium of the game, which means in the prisoner's dilemma it would discover the incentive to cooperate.\n- The paper reads like a (still relatively interesting) collection of examples rather than a \"unified\" research paper. I think the paper would gain from having a unified model for ADS that all the current examples in the paper derive from. It would also be useful to have a centralized definition for the unit tests it uses for example. Right now, I find the unit tests very ad-hoc and case-by-case, and I think the paper would benefit from defining them in a more principled way.", "summary_of_the_paper": "The paper looks at the problem of auto-induced distributional shift, which is what happens when the inputs to a machine learning algorithm are affected by the algorithm used for learning itself. In general, ML algorithms do not face an i.i.d. distribution of data or users; rather, in recommendation systems for example, the choice of content displayed will affect how users react to it and even which users we face. In machine learning applied to users or humans, said users/humans we make decisions on may try to react to the decision rule in order to obtain better outcomes. The authors aim to study ADS, and to design tests for incentives, to detect whether given algorithms hide or detect incentives that cause ADS.", "summary_of_the_review": "I think the paper has some interesting ideas, but I think it needs to be improved before it is above the bar for acceptance at ICLR. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "I1ZG4L1do8", "reviewer": "Reviewer_C74h", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is rich in content and references. I like the idea of using POMDP to simulate the characteristics of a self-selection loop, as well as the novel adaptation of the prisoner's dilemma. There are other interesting ideas like the close examination of meta-learning and q-learning.\n\nHowever, my main concern of this paper is the lack of validation in real-world scenarios. For example, could the authors collect data to match the POMDP simulations with real-world events? As the paper proposes tools to study the techno-societal impacts of technology, having empirical validation seems like an important step. The two simulations in the paper alone do not show clear evidence of the generality of the proposal.\n\nThis concern is compounded with my lack of intuition of why we must \"hide the incentives\" or, as I understand it, completely block the gradient path, between a state change and the final reward. As the authors suggest as well, having cooperation between the environment states and the agent actions is important in many POMDP problems. For example, we cannot ask a recommender system to not have an influence on the society. Rather, we should be asking how a recommender system should influence the society. I.e., we may want more interpretability in the information leakage than a complete blockage. The oversimplification of the problem leads to lesser impact.\n\nOn the technical side, I have some confusions on Experiment 1. It seems that the agents never get into the \"cooperate\" state with more than 50% chance in global probability, when the gradient path is blocked between the current and the future states. It becomes hard to tell whether ADS happened or rather the agent model simply failed to converge. Is it possible to produce near-100% \"cooperate\" rates in the given setting? I also have problems following the explanations for the \"cooperate\" result. It seems that this is due to some kind of gradient leakage in the outer loop of meta-learning, but I am not sure. I also do not have strong intuitions in the analysis of the q-learning algorithm.\n\nExperiment 2 is pretty much self-evident. In the pursuit of optimal strategies, the agent policy, as well as the state visitation, will eventually saturate around the optimal choices. This concentration effect is exactly why a mountain car agent can successfully move up the valley through a set of particular actions to receive the highest rewards, given sufficient reinforcements. In RecSys, I would not completely dismiss the concentration effects. For example, in product recommendation, the concentration leads to efficient goal-oriented shopping experience. On the other hand, for content recommendation, the concentration could be bad - but this is often compensated by the uncertainty of new contents ingested in real-time. As a result, people still consider Fox/CNN to be valuable sources of information despite their potential biases. I would love to see more discussions on the complex nature of the bias problem.\n\nSome style comments:\n- [abstract] \"whether an algorithm will hide or reveal incentives\" - Missing a lot of contexts to understand this sentence on first pass.\n- [abstract] \"correctly diagnoses its propensity to reveal incentives to ADS\" - Seems like an over-claim.\n- [intro] \"2. Changing the distribution of users such that predictions are easier to make\" - RecSys do not change the distribution to make the predictions easier to make. They just happen coincidentally.\n- [intro paragraph 2 middle] \"quantify the desirability of all possible means\" - Means and meant in a few lines above cause confusions.\n- [page 2 paragraph 3] \"one that does not induce ADS; the other is hidden and we want it to remain hidden\" - fragmented sentence.\n- [contribution Point 2] \"liable\" - replace with simple words.\n- [contribution Point 3] \"confirm qualitative features\" - hard to parse.\n- [Page 3 PBT] \"because they give the OL more control than many other meta-learning algorithms\" - clarify what other meta-learning algorithms this is referring to.\n- [Page 3 second-to-last paragraph] \"self-refuting predictions\" - use simple words\n- [Page 3 second-to-last paragraph] \"Leike et al., 2018\" - does not seem to talk about news manipulation\n- [Page 3 last paragraph] - there is a lot of missing context. what is i.i.d.? how is RL environment different from the i.i.d. assumption? \n- [Section 4 first paragraph] \"the incentive to drive users away\" - there is never an incentive to drive users away in a RecSys. It just happens.\n- [Page 4 second to last paragraph] \"of influencing\" - to influence\n- [Page 4 last paragraph] \"Everitt & Hutter 2018\" - 2019.\n- [Page 7 (2) exploiting correlation] - I do not quite follow this paragraph\n- [Page 8 Section 5.2.1] \"the covariate shift rate (alpha_1) is faster than the concept shift rate (alpha_2)\" - are they defined earlier?\n- [Section 5.2.1] - where are the results for context switching?\n- [Page 9 Safety and incentives] \"the efficacy of this approach of hiding incentives and ways in which it can fail\" - too many concepts in one sentence. not sure what the \"ways to fail\" refer to.\n- [Page 9 Conclusion second paragraph] \"affect their propensity to reveal incentives for ADS\" - propensity => likelihood\n\nEdit:\nI second Reviewer yBdS in the suggestion that no-regret algorithms in the outer loop could in principle \"memorize\" the aggregated information from the inner loop.", "summary_of_the_paper": "The paper talks about a self-selection phenomenon which the authors call Auto-induced Distributional Shift (ADS). This phenomenon is common in recommender systems, where the promotion of a type of contents (e.g., liberal versus conservative) leads to a change in the active user base. Self-selection bias is a big topic and hard to quantify precisely. Instead, the paper makes a novel connection to Partially-Observable Markov Decision Process (POMDP), where the self-selection effect is analogous to the changes in the environment states (i.e., the active user base) after agent decisions (i.e., content recommendation). With this setup, the paper is able to simulate the self-selection process and analyze the system parameters that may affect the characteristics of the process, such as its mixing time.\n\nThe paper also contains some other discussions on whether we can \"hide the incentives\" or, as I understand it, completely block the gradient path, between a state change and the final reward as a way to prevent ADS. This is substantiated by a self-cooperating/defecting game adapted from prisoner's dilemma and a comparative study of a few learning algorithms including meta-learning and q-learning. This seems to bring novelty, but I have some questions about the technical details as well as how to apply the ideas in more general cases.", "summary_of_the_review": "I like the idea to simulate self-selection bias with POMDP. However, I have questions in the discussions of the comparative studies (ADS effects as the authors explained versus simple failures of convergence). I also question the proposal to completely block the gradients to discourage taking advantage of any changes in the environmental states. I gave counter examples in RL mountain car problem, goal-oriented online shopping, and even news recommendation when opinion biases do not stop it from providing useful information. The paper's proposal seems oversimplified and it needs to be validated in real-world scenarios to produce sufficient impacts for this venue.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to study Auto-induced Distribution Shift (ADS), the phenomenon that models can create a feedback-loop: the predictions of a model influence user behaviors when it is deployed, which, in turn, affects the accuracy measure of the model. The paper empirically shows that a meta-learning algorithm called PBT causes a distribution shift instead of maximizing accuracy. While the premise of this paper is interesting, the proposed frameworks are very similar to the idea of strategic behavior in machine learning, and of "Performative Prediction" (Juan C. Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, Moritz Hardt). However, this line of work is neither cited nor discussed in this paper. In addition, the paper is hard to read in certain parts. We encourage the authors to compare their work with performative prediction. We hope the authors find the reviews helpful.
Decision: Reject