Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper discusses concepts of self-induced distributional shift (SIDS) and the hidden incentives when using meta-learning algorithms. It then prescribes a unit-test to check whether there is hidden incentive for distributional shift (HIDS) in the algorithm and proposes to use context swapping to mitigate such phenomenon.

I am not very familiar with this series of research but I am wondering why the paper focuses on meta-learning and its connection to HIDS. Does it use meta-learning as a tool to identify HIDS? But from the description and experiments, it seems the paper is talking about how meta-learning itself leads to HIDS, for example, by comparing different hyper-parameter setting for meta-learning and PBT, it shows the unit-test is failing. So it seems like meta-learning itself leads to distributional shift?

Also I cannot fully appreciate the utility of this "unit-test". It is well known normally an interactive system that can change its inputs have distributional shift. What other information does this "unit-test" inform us? Similarly, how does "context swapping" mitigate distributional shift? From the experiments, it dumbs the meta-learning algorithms and make it pass the "unit-test", but I am not sure what other practical benefits it can bring to improve real systems.

Usually, a reinforcement learning algorithm can meaningfully mitigates the adverse effects of distributional shift by explicitly modeling this interactive process and evaluating rewards with considerations to distributional shift caused by different policies. It is difficult to see how the concepts discussed in the paper provide meaningful approaches to address the issue.

Overall, the paper touches the important question of distributional shift for machine learning systems but I find the concepts discussed in the paper, such as the focus on meta-learning, the "unit-test", and "context-swapping", less relevant to how we can really mitigate the issues in real systems or how it can provide additional insights about the problem.
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The main idea of the paper: When using meta-learning there is an inherent incentive for the learner to win by making the task easier. The authors generalise this effect to a larger class of problems where the learning framework induces a set of Hidden Incentive for Distributional Shift (HIDS) and introduce Context Swapping, a HIDS mitigation technique. In the experimental section, the authors propos a HIDS unit test which then they employ to show that PBT (Population Based-Trainng), a popular meta-learning algorithm exhibits HIDS ant that context swapping helps fixing it. 

Overall, I found the idea of the paper interesting, but the attempt to generalise the effect from meta-learning to general learning setups hard to follow and detracting from the overall value. I think the authors should have restricted their claims to PBT and PBT-like methods and follow-up with something more general in future work. 

Furthermore, the notation and formaliation of the problem are incomplete:
    * the concept of ‘trajectory’ is introduced without being properly defined, though its crucial in the definition of the proposed HIDS mitigation approach
    * the context swapping algorithm description is not clearly motivated and explained, a diagram showing the learner shuffling would be quite helpful
    * in the HIDS unit-test section, the game theoretical setup is only partially explained, the defection and cooperation actions are not clearly linked to the HIDS 
    * In Section 4.1.1 HIDS UNIT TEST EXPERIMENTAL RESULTS AND DISCUSSION Figure2 is refered for results without ever stating the task and the Figure itself does not mention it

       In terms of suggestions, I think the paper needs to go through a careful refactoring with an attention to technical details (careful concept definition, introduction of notation, clarity on experimental setup) 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors study the phenomena of self-introduced distributional shift. They define the term along with the term hidden incentives for distributional shift. The latter describes factors that motivate the learner to change the distribution in order to achieve a higher performance. The authors study both phenomena in two domains (one being a prisoner dilemma and the other a recommender system) and show how meta-learning reveals the hidden incentives for distributional shift. They then propose an approach based on swapping learners between environments to reduce self introduced distributional shift.

In my opinion this paper should be (weak) rejected for several reasons.

-	The paper is poorly written. Some sentences are hard to comprehend, even after repeated reading. It feels hastily written and should be carefully proofread with the help of a proficient English speaker.
-	The very first paragraph is not a helpful example. While the authors provide an example that that illustrates a distributional shift, they don’t describe what the shift is and why that shift in a negative consequence for learning. Therefore, the example is not helpful, but rather confusing.
-	In point 3 of the contributions, the authors state ‘that meta-learning reveals HIDS in these environments’. Is this true for all meta-learning approaches? The current statement overclaims their findings. If the authors decide to keep it, they should provide a description of the types of meta-learning approaches that reveal HIDS and which don’t or evaluate all different meta-learning approaches.
-	Section 4.1 is hard to follow. The authors state that the meta-learner is used to tune the learning rate – they fail to clearly explain how exactly the learning rate tuning results in a non-myopic behavior for a myopic learner.
-	In the Q-learning example in Section 4.1, is the effect of \epsilon considered? The discovery of non-myopic strategies might simply be based on chance. I would like the authors to include an investigation into the effect of this parameter.

Generally speaking, I appreciate the author’s thoughts on SIDS and how they approach revealing hidden incentives. Although I vote to reject this paper, I strongly encourage the authors to rewrite the paper, address all other issues that are noted during this peer review and resubmit.

review_title: Official Blind Review #2

Meta-review: The paper shows how meta-learning contains hidden incentives for distributional shift and how a technique called context swapping can help deal with this. Overall, distributional shift is an important problem, but the contributions made by this paper to deal with this, such as the introduction of unit-tests and context-swapping, is not sufficiently clear. Therefore, my recommendation is a reject.
Decision: Reject