Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper discusses a phenomenon where machine-learned models may influence user behaviors in future iterations, creating self-selection effects such as filter bubbles or propagation of fake news. The paper calls these effects auto-induced distribution shift (ADS) and argues that a specific meta-learning algorithm PBT manipulates users instead of maximizing rewards. To illustrate the ideas, the paper introduces a few simulation environments where user behaviors may change when certain prediction events happen.

Despite an interesting motivation, the paper is quite vague in the proposed methods. For example,
1. I am not able to associate ADS with the specific meta-learning algorithm called PBT. Self-selection is a universal phenomenon present in many types of algorithms. It can be corrected by any learners that focus on the detection of distribution shifts, e.g., via inverse propensity scoring, or under ignorability assumptions. On the other hand, PBT is just an algorithm with automated hyperparameter search. What can we learn by connecting ADS to PBT?
2. Why are reinforcement learning / supervised learning free from incentive / ADS issues? For the reasons above, I believe these claims are unrelated and wrong.
3. Is learning conducted online or offline? Would user states be observable or unobservable? If offline, how would the learner foresee the shifts in user behaviors after online feedback loop? If online and observable, shouldn't we focus on modeling user state transitions? If online and unobservable, what are the belief states?

Additionally, the experiments are far from being practical. While Experiment 1 makes sense that by sacrificing an immediate reward, the user gets "manipulated" into a state of more predictability, I am not seeing how Experiment 2 makes practical sense. In this example, the environment is set as a modified prisoner's dilemma. The paper assumes "defect" to be a better solution, but in this specific case, it seems that cooperation seems to be the true user utility, despite the user gets "manipulated" by non-myopic RL algorithms.

The paper also lacks general clarity. I am not sure how content swapping works in Section 4.2. The details about Q-learning should be reorganized. Section 5.2 is best presented with equations or diagrams. Overall, the authors should think about how to eliminate the unrelated claims and simplify the paper to include just one proposed method, which solves a wide class of interesting problems at the same time.
review_title: Interesting idea, but the proposal lacks clarity

final_score: rating: 6: Marginally above acceptance threshold
review_content: Abstract: The paper highlights an interesting problem of learning dynamics, where a learning system has the incentives to change it’s future input in order to increase its performance. This is not always problematic, but it can, at times, lead to perverse incentives for the system, such as under-performing on users with complex profiles in the present, in order to remove them from the future testing sets. The authors propose a set of unit tests to detect this issue and a way to solve the problem for the PBS metalearning frameworks. The experiments on toy/simulated datasets show the relevance of the approach.


Pros: 
- Clarity: The paper is quite well-written and the explanations are clear. I think the problem that the paper aims to address is real and of practical value.
- Experimental design: The experimental section, is very clear and supports the claims of the paper

Cons:
- Significance/Impact: I find the scope of the paper to be limited to Population Based Training, which as far as I can tell, its not a well-established training method in real-world applications, especially recommendation. I would like to see the expansion of the same ideas to general batch learning from bandit feedback which is as far as I can tell the most established setup for RW ML decision systems.
review_title: Good idea, really interesting problem, but however of limited impact due to its close relationship with Population Based Training

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces the concept of auto-induced distributional shift (ADS), and argues that some meta learning and reinforcement learning algorithms have the incentives to change the distribution so that the problem is easier to solve. The paper presents unit tests to detect hidden incentives for auto-induced distributional shift (HI-ADS) and also proposes context swapping to reduce the distributional shift. Experiments on Population-Based Training (PBT) show that PBT reveals HI-ADS in unit tests and context swapping mitigates the distributional shift for PBT.

Pros: 
1) The concept of ADS is interesting. Distributional shift is not a new phenomenon especially for reinforcement learning. As mentioned in the paper some existing works try to exploit the distributional shift. And if distributional shift is harmful then algorithm could use a better reward function to mitigate the effects. But this concept is indeed not formally defined and studied in previous works.

2) The results of unit tests reveal hidden incentives of PBT under two toy environments. 

I have the following concerns and questions:

1) How to measure HI-ADS is unclear. From the definition of HI-ADS in Section 4.1, it seems that we should evaluate by whether "the learner would not learn to perform the incentivized behaviors at higher than chance levels".  In the unit test, the incentivized behaviors are clearly defined for the two toy environments, but the paper does not discuss how to find / define incentivized behaviors for a general environment. This directly leads to my second concern.

2) How to generalize the unit test? The unit test only tells whether an algorithm reveals ADS in the two toy environments. Even if a meta learning algorithm passes the unit test, it is not guaranteed that such algorithm won't exhibits ADS in other application scenarios. Can we get more from the unit test? In real world applications if we do not have a perfect understanding of the environment that what kind of behaviors are encouraged by incentives, then how to evaluate HI-ADS?

3) It is not clear whether PBT algorithm itself or all meta learning algorithms would exhibit HI-ADS. The authors use PBT as an example to illustrate HI-ADS, but it is unclear whether this observation suggests other meta learning algorithms are also vulnerable to ADS or that we should test them individually using the unit tests. 

4) Why would context swapping mitigate the distributional shift? It is not clearly explained. My rough understanding is after switching the environment, the learners are using misspecified models and won't fully exploit current estimation.  Does context swapping help with content recommendation task? Is this a general approach for other environments instead of the toy unit test setting?


Other comments:

1) Figure 5 and 6 are too small to read. 

2) Appendix is missing, which is supposed to contain additional details on experiments according to the main content.



review_title: An interesting concept, but not clear in many aspects

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes to study auto-induced distribution shift (ADS), a phenomenon where the machine learning model itself may change its own data distribution via its decisions or actions. The paper seeks to understand, in situations where ADS is undesirable, whether learning algorithms are privy to, and pursue, incentives for learning models which take advantage of ADS.

Pros:

+ The paper proposes an interesting and promising direction for study.
+ Simple examples serve to illustrate the concepts outlined in the paper.

Cons:

- Both expositionally and experimentally, there isn't a compelling argument made in terms of when ADS shows up in real problems.
- There is limited insight as to how the problem of interest may be addressed methodologically.

Weighing these pros and cons, I am inclined to reject this paper. More details are provided below.


Quality
---

Perhaps what detracts from this work the most is the lack of rigorous evaluation on real world problems. It is not difficult to imagine that ADS may indeed be a problem that must be dealt with in certain applications, but it is also unsatisfying to leave this to the imagination. All of the experiments presented are toy, and though toy experiments are certainly valuable for gaining insights into the problem, they should be accompanied also by larger studies that corroborate the story.

The content recommendation experiment seems promising and could perhaps be built out into a larger scale experiment. But this raises additional questions about why the proposed method of context swapping is not effective at handling this situation. In general, the context swapping approach seems rather ad hoc, and might there be potential downsides regarding tradeoffs in model performance? Further investigation into this or other mitigation strategies would also significantly strengthen the work.

Clarity
---

The paper is generally well organized and well written. I appreciate the hierarchy of definitions as well as the discussion of how certain frameworks such as meta learning can exacerbate the problem being studied.

Originality
---

I am not an expert in this area, but the paper seems to propose an original problem to study, and discussion of related work seems relatively well researched and well written.

Significance
---

Relating back to my previous comments, the exposition needs to do a better job at convincing the reader that the problem is important. Further discussion into some of the prior works mentioned in the related work section would help in this regard. Is it possible to characterize prior work into the taxonomy and framework described? Might it even be possible to design (or borrow) an experiment based on this prior work?

In my opinion, the current exposition is not enough to convince the reader that the problem is important. The running content recommendation example seems hypothetical, and a real world system would likely, perhaps even obviously, disincentivize such behavior from a model that drives users away. This indeed may be a hand designed solution, which as the authors correctly point out may not scale to cover all instances of this problem, but it is up to the authors to argue this more convincingly.
review_title: Official Blind Review #1

Meta-review: This is a thought-provoking paper which describes a significant problem that plausibly occurs in deployed ML/RL models.
The paper is clearly written, describing claims using examples and developing small unit-tests to probe models.
However, as the reviews and discussion show,  the exposition should be substantially re-worked so that the core contributions are more understandable -- the core message in the revised manuscript is still very nuanced and easy to mis-understand.

Let's say we train an ML model using supervised learning to minimize a loss function on a dataset. Several models may have near-optimal loss as measured on a validation set -- a learning algorithm is free to return any one of them. Now in a deployed system, these ML models are not merely generating passive predictions; these predictions are driving system operation and potentially influencing future states/contexts/inputs that the model will be invoked on. It is well known that supervised learning makes an iid assumption between training and deployment which is violated in this setting -- that is not the main point of this paper. Consider again the set of models with near-optimal loss. Some of them, when deployed, may cause the distribution mismatch between training and deployment to be miniscule, while other models may introduce a vast mismatch. We may choose a learning algorithm which just so happens to pick models from the former category; and we may conclude that feedback effects induced by the ML model are not substantial. We then change some unrelated detail in the learning algorithm (but not the objective, datasets, validation criteria, etc.) which just so happens to pick models from the latter category and suddenly witness a large distribution shift. What happened? And could we have developed tests to detect that our learning algorithms have these tendencies? The paper attempts to articulate such questions, and design the first step in answering them.

Moving to RL, where we routinely consider distribution shifts in states visited by different policies, does not fundamentally fix all these issues because the reward function is typically an engineered proxy to elicit desired behavior -- and we may again find that some RL algorithms have a tendency to find reward-maximizing policies that exploit gaps in reward specification as opposed to following intended behavior.

The core question studied in this paper, scoped to the supervised learning setting, is very related to that of strategic classification (see e.g., https://arxiv.org/pdf/1910.10362.pdf Strategic Classification is Causal Modeling in Disguise). The following sketch is inspired by that literature.

We might hope to augment the training objective of ML/myopic RL/strategic RL to address the Auto-induced distribution shift problem as follows.
[Supervised learning for content recommendation] Let the training/validation data distribution be D. Assume for now that there is no exogenous factor in the environment that causes any distribution shifts in deployment -- so, the only shift is due to feedback effects from the predictions made by the model. For an ML model f, let the corresponding recommendation policy be pi_f, and let the long-term distribution of data seen from user-interactions with pi_f be D[pi_f]. Then, what we want is: f* = argmin_F Expectation over D [ L(f) ] subject to constraint that D ~= D[pi_f]. 
For a contextual bandit/myopic RL formulation of the problem, we could similarly constrain the learning problem as pi* = argmax_Pi Expectation over D [ Reward of pi] subject to constraint that D \approx D[pi].
Essentially, both supervised ML and contextual bandit algorithms are assuming that context distribution is unchanged -- so let us enforce that the context distribution is indeed unchanged as a consequence of the policy's actions.
It is unclear how to generalize this kind of thinking to situations where environmental changes also contribute to distribution shift.  The authors call out precisely this flaw using the cryptic comment -- 'not trying to change X' is not the same as 'trying to not change X'. The formulation above does 'trying to not change X', but that is an insufficient band-aid in situations when environment changes X. It's also unclear how one might estimate D[pi] or D[pi_f] and appropriately constrain the learning algorithm -- but these are all interesting questions to study.

The paper in its current form is asking an important question. In supervised learning, the desired solution might actually coincide with strategic classification solution concepts. The paper may be asking a generalization of the phenomenon for myopic RL and RL. It may spark interesting discussions and follow-up work, but is not yet mature beyond a workshop poster.
Generalizing the unit-tests, articulating the scope of situations where context-swapping may be a useful strategy, and even formalizing the problem and desired goal (as attempted above for the content recommendation example) will substantially strengthen the paper.
Decision: Reject