Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new meta-RL algorithm containing two main components: 1. Model Identification 2. Experience Relabeling. Model Identification models the next state and reward function condition on the previous state, action, and context.  Experience Relabeling module uses the model learned in the previous step to generate synthetic transitions to improve policy learning. Results on in-distribution tasks are comparable with the previous method and it shows good performance on the out-of-distribution task.

1. Paper claims on page 4 second paragraph that the main difference between context variable in this paper and previous works is that gradient descent is used to adapt the context. That is not entirely true as previous work like MQL does use gradient descent to adapt the context (MQL should be cited there too). Am I missing something here?

2. On page 3, the second paragraph from the bottom, you claim that "a sufficient condition for identifying the task is to learn the transition dynamics and the reward function, and this is exactly what model-based RL methods do." If that is true, why did you use context? I don't think this method will work without context (an ablation study would have been nice with/without context). 

3. I am not sure if I fully follow your argument about gradient-based meta-learning with value-based in Appendix A or at least it is a very weak argument about the relation between Q function convergence, bellman backup iteration used in actor-critic, and gradient descent.  Can you clarify a bit?

4. Can you explain how Experience Relabeling helps with out-of-distribution tasks? As far as I understood from your paper, model(\hat(p)) is trained using **current** tasks distribution and this model will be then used to generate synthetic data. Then how come it can help with out-of-distribution tasks when it only saw in-distribution tasks? 

5. Why for in-distribution experiments, Experience Relabeling was removed? experience relabeling should be as useful as for in-distribution tasks as for out-distribution ones.

6. Do you **only** update context parameters in Eq. 1? not \theta? if that is true, figure 1 is very confusing as it implies the model( \theta) and context will be updated together. 

7. How context is constructed? is it a recurrent network? or? 
  
8. Is it really required to use MAML regression to learn the model? wouldn't be enough to just use simple regression to learn the model?

9. This paper writing needs improvement especially sections 3 and 4 as they don't have a smooth flow. It made me read these two sections multiple times to understand this MIER.  And introduction needs improvement too.

10. Utilizing other benchmarks like meta-world [https://meta-world.github.io/] will be more useful than the current benchmarks to evaluate this method per MQL findings.

11. The paper would have been stronger, if the authors had provided ablation studies in which they closely study different component of their method (w/wo context, simple regression to build model, show who close or far generated data from actual out-of-distribution tasks, etc.)

12. Results in this paper are neither that convincing nor that strong. This method only shows good performance on Cheetah-Negated-Joints and Ant-Negated-Joints on out-distribution.  In addition, I don't understand why this method can't have better performance on in-distribution tasks.  Model + context should be more expressive than just using context ( e.g. MQL vs. this method.)


------- Update after rebuttal ---------

Thanks for your response. Even though your responses clarified some of my comments, I still don't understand how Experience Relabeling can help with OOD and why your method doesn't good enough with in-dist data. As as result, I stay with my current evaluation and score.
review_title: my review 

final_score: rating: 4: Ok but not good enough - rejection
review_content: - Summary: this paper proposes a novel meta-RL method that works as a 2-stage pipeline (with the 2nd stage being optional). The first stage does model identification by having a network doing meta-SL on a learned model that predicts transitions and rewards. This model uses a MAML-like algorithm to adapt a context variable that is passed to a policy network. This policy network does regular RL taking the context variable (describing the task) and the states as inputs. This approach alone would work well in regular meta-RL, but we aim at robustness under distribution shift, for which we need the second part: experience relabeling. There we use the adapted model from the first part to relabel the data coming from other tasks in meta-training and continue training the policy on that synthetic data.
- Pros:
    - 1. The method is nice, simple and well-founded.
    - 2. We're turning a meta-RL problem into a meta-SL + RL problem; that's a nice decomposition.
    - 3. Generalizing outside of the meta-distribution is useful and making the approach consistent is a good requirement 
    - 4. The paper is well-written and clear and the experiments are well-executed.
- Cons:
    - 1. I think the experience relabeling shouldn't work well whenever the state distribution changes substantially and thus we haven't seen the states at meta-training time. For instance, in the ant domain where we go to a corner that has never been explored, there shouldn't be much states to relabel in that portion of the state-space (the experimental results in that particular experiment are also worse than MQL). That defeats the extrapolation purpose of the relabeling and should be discussed further. This issue needs to be discussed further. W.r.t. this point, at the end of section 4 it is claimed that MQL will perform worse when the data support is different because MQL assumes it's the same support, which feels similar to what I'm saying; however, then I don't understand why MQL performs better in Ant-Dir (the example domain I gave).
    - 2. The paper mentions the proposed approach being composed of "two independent novel concepts", but each has been studied before: model identification is common in controls and experience relabeling has the same feeling as Hindsight Experience Replay ` https://arxiv.org/abs/1707.01495 ` or the off-policy corrections of this work ` https://arxiv.org/pdf/1805.08296.pdf .`
    - 3. The experimental results are underwhelming: in the in-distribution the results are roughly on par with PEARL and MQL; which is to be expected since that's not the main contribution of the paper. However, the out-of-distribution tasks results are quite underwhelming, given that this is the only algorithm with out-of-distribution in mind: the proposed approach is only significantly better than MQL in 2/5 and statistically worse in 1 and also only better in 1/5 than its ablation MIER-wR.
- Clarity: high
- Significance: medium
- Originality: medium-low
- Questions:
    - Negated joints means that putting 'a' in that joint is the same as putting -a if the joint was not negated?
    - My understanding is that we are _not_ backproping the policy training back to $\phi_T$ and then back to the model inner loop training that produced $\phi_T$. Is that correct? If we don't, wouldn't it make more sense to do so?
- Note:
    - The explanation of appendix A was useful, thanks for including it.
- Reasoning behind decision: this paper proposes combining two relatively well-established ideas in a novel and potentially-useful way. If the extrapolation experiments were more convincing I would lean towards acceptance; but, as this is currently not the case, I am hesitant to endorse it.

=============
UPDATE POST DISCUSSION

Thank you for your responses.
- I am more convinced about the novelty of the proposed methods.
- I am still unconvinced about relabeling being the best we can do for out-of-distribution state distributions. Building learned methods that generalize more broadly or many-inner-step meta-learners could help quite a bit.
- **I got concerned that a question by AnonReviewer2 "Why for in-distribution experiments, Experience Relabeling was removed?" went unanswered.** Having evaluated MIER-wR, adding MIER to the plots during rebuttal should have been very easy. Moreover, in practice, we shouldn't be able to leverage the fact that we know a task is in-distribution or out-of-distribution and choose between MIER and MIER-wR accordingly (which would also be a bit ugly). Therefore, in my opinion, putting MIER-wR as a representative of the paper instead of MIER is not valid.

Since most of my concerns pre-discussion are still there and we cannot assess whether MIER does well on in-distribution (and MIER-wR is not better than MQL for OOD tasks) I, unfortunately, have to decrease my score from 5 to 4.

- This prompted me to take a closer look at the actual numbers of the plots and I got more concerned. I didn't take it into account in my rating update because I should have brought this up before the discussion: 
   1. I feel something is off with PEARL vs MIER-wR when looking at in-distribution vs OOD in half-cheetah vel. In particular, we see PEARL does considerably better than MIER-wR on in-distribution (Fig 2), but its performance starts already much lower than MIER-wR in Fig 6 left. 
   2. Another concern is whether the numbers for OOD are "decent" in the sense of creating meaningful policies. Half-cheetah-vel-hard has a reward of ~-325, which is lower than MQL with 0 data for in-distribution half-cheetah-vel. This may be because the evaluated tasks are different, but raises concerns that maybe all baselines and MIER are all doing poorly.
review_title: Nice algorithm but underwhelming results

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper presents MIER, a novel meta-RL method that is designed specifically to be able to deal with out-of-distribution test tasks. The idea is to learn two separate modules: a dynamics/reward model that consists of model parameters $\theta$ and context parameters $\phi$. This model is trained using a MAML-like gradient-based supervised meta-learning algorithm, and updates the context vector $\phi$ using gradient descent per task (and the initialisations of $\phi$, $\theta$ are meta-trained), together with a policy that is conditioned on the "context vector" $\phi$, which is task-specific after the gradient update. In order to adapt to out-of-distribution tasks at test time, the dynamics/reward model can (1) be trained with the new data, updating both $\phi$ and $\theta$, and (2) be used to train the policy using data obtained from he new model.

Impression:
The problem setting of out-of-distribution adaptation in metaRL is very interesting and relevant to the community. I think the idea of separating the model and the policy makes a lot of sense, since the model can be used to generate synthetic data on a new task for training the policy. I'm a bit underwhelmed by the empirical evaluation, and on the one hand I have a lot of open questions about what MIER is capable of doing, and on the other hand I do think that MIER *can* potentially do quite interesting things but the experiments in the paper don't really show this. I find the "extrapolation over dynamics" experiments most interesting. Here the MIER method really shines compared to other approaches. I think the paper could be made stronger by investigating the properties of MIER on a wider set of interesting experiments, and really analysing what it does and why it works well for certain problems (see concrete questions/suggestions below). I also feel like the adaptation to out-of-distribution tasks is a bit heuristic (see question below) and it's unclear to me how this would work on other / real world problems. Overall, I think this method is of interest to the community, but I also think the impact could be much higher with more discussion and empirical analysis. Hence this is a borderline paper for me at this moment.

Questions:
1. In practice, you would have to determine whether or not to update $\theta$ in addition to $\phi$, and for how many gradient steps. 
 a. How do you do that, and how can you prevent under- and over-fitting? 
 b. Since $\theta$ isn't trained to adapt quickly to new tasks (unlike $\phi$), could it happen that with little data this actually is a really bad initialisation and I won't be able to appropriately adapt? So even though MAML is consistent, this doesn't really help me if I don't have a lot of data.
 c. If you're looking at an out-of-distribution task, do you update both $\phi$ and $\theta$? Do you update both at the same time, or first $\phi$ and when that's not enough you update $\theta$? I guess you'd want to make sure that *most* of the new info goes into $\phi$ since that will make training the policy with the learned model easier. Is that true?
2. The pre-update policy is responsible for data collection. I expect this to be somewhat random, which works well in settings where even random rollouts contain enough information about the task. But what if that's not the case? I can see this becoming an issue in settings with sparse rewards. Do you have a sense for the exploratory behaviour that the pre-adaptation policy exhibits when collecting $\mathcal{D}_{adapt}$? In some follow-up papers to MAML (ProMP https://arxiv.org/abs/1810.06784, E-MAML https://arxiv.org/pdf/1803.01118.pdf) it has been shown that the pre-adaptation behaviour in standard MAML can be quite poor. In the environments you evaluated MIER on, the rewards are so dense that even on out-of-distribution tasks, a somewhat random pre-adaptation policy can gather data that is useful to update the model. How would you handle cases where this is no longer the case (e.g., sparse reward settings or environments that require better spatial exploration to learn about the task)?
3. In the abstract you write "Current approaches are either not able to extrapolate well, or can do so at the expense of requiring extremely large amounts of data for on-policy meta-training". Can you explain this a bit more? Why is an on-policy more likely able to extrapolate well compared to an off-policy method? Do you have examples of methods where this can be seen empirically? 
4. What is the end-performance of RL2, MAML and ProMP when trained for longer? I agree that sample efficiency of MIER is a bonus, but I think we would get a better sense of how the model works and compared to baselines if we let those baselines converge.
5. I disagree with some statements about model consistency: 
 - Paragraph after Eq 2: "the model will still adapt to out-of-distribution tasks given enough samples and gradient steps, since the adaptation process corresponds to a well-defined and convergent learning process."
 - First paragraph of section 4: "although the gradient descent adaptation process is consistent and will continue to improve (...)".
 - The thing I disagree about is that updating $\phi$ only is not a consistent learning algorithm, because $\phi$ is only an input vector, and might not be expressive enough such that a completely new task can eventually be learned. So to my understanding, this argument only holds for $\theta$ (+$\phi$). Could the authors comment on this?

Suggestions:
- It would be nice to add numbers to all equations, not just the first one because this makes it easier to talk about them. In my review I refer to equation numbers by just counting them.
- Sometimes I'm a little bit confused what you mean when you write "model". It's often the dynamics/reward model, but it's easily confused with the policy (e.g. last paragraph in introduction), and it's also not always clear if you refer to $\theta$ or $\phi$ (e.g., second sentence after Eq 2;  2nd+3rd sentence on page 5; and when you talk about model consistency). The readability of the paper can be improved by making this more explicit by saying "the dynamics model" or "the model parameters $\theta$" where appropriate.
- Zintgraf et al (2019) show that you don't need to meta-learn the initialisation of $\phi$, but you can always set it to to $0$ before the inner-loop update, which saves memory and is easier to implement. I can imagine that this helps with learning the policy because the values of $\phi$ might fluctuate less during meta-training. The policy that collects $\mathcal{D}_{adapt}$ then also *always* gets $\phi=0$ as input, which again could stabilise training. It might also be a way forward to look at ways to learn a good exploration policy (because now it's easier for the policy to *know* that it hasn't gathered any information yet). Might be worth a try!
- I found it weird that you compare to GrBAL on one of the environments, since it's much worse anyway. If you add this method, then it would help to explain why you compare to it, and what you can take away from this comparison. (And if there's no take-away, consider moving it to the appendix because it just distracts otherwise.)
- The experiments section has little discussion about why MIER works the way it does, in comparison to other methods. Especially on the "extrapolation over dynamics" section, where MIER really shines, a more in-depth discussion or even analysis of what is going on would really strengthen the paper, and help the read understand the method. E.g., for the "shared support assumption hypothesis" that you mention compared to MQL - can you design an algorithm where you explicitly test this?
- It would also be interesting to add an analysis / plot on how good the dynamics model is doing when it is updated on a new task, and how much data it needs to learn this. 

--------------------------------------------------------------------------------------------------
UPDATE

I read the other reviews and the author's responses. Thanks for replying to my questions, that made some things clearer!

Some final comments: 
- "We performed a hyper-parameter search to find the best settings of the steps to take on $\phi$ and $\theta$ for each of the environments. To ensure that we don’t over-fit the model when adapting for extrapolation, we use 80% of the available test-data to train the model, and use the rest for validation." - If I understand this correctly you tune the number of gradient steps on $\phi$/$\theta$ on part of the out-of-distribution adaptation data, and evaluate on the rest. That feels somewhat restrictive to me and might not always be possible - in which case your results are a bit optimistic. In the response to R1 you wrote that "the knowledge of in-distribution or out-of-distribution is not necessary to apply our method", but in this case the question is how to determine how to decide how often to adapt $\phi$, then $\theta$, before training the policy. At the very least, this should be discussed in more detail in the main part of the paper, given that this is central to your method. 
- Thanks for pointing me to the appendix of the paper that has the end-performance of your baselines. However I do think these should be included in your paper in order for the reader to get the full picture. There might be differences in implementation (as an example, some papers use different horizon lengths in MuJoCo) so it's not always possible to compare numbers across papers, and your paper should be self-sufficient. 
- Not meta-learning the initialisation of $\phi$ (but set it to zero before every inner-loop update) seems like an important detail to me - this means in Eq (2) you don't actually take the gradient w.r.t. $\phi$. It might also be an important to stabilise policy learning (because $\phi$ doesn't change too much over the course of meta-training). Even if it is an implementation detail, it should at least be mentioned in the appendix so that somebody who wants to re-implement your method can reproduce results!
- Using something like [2] for good pre-adaptation exploration is a good idea, I agree! (And I agree it's out of scope, but might be worth mentioning in the paper.)

I think overall the idea is promising but the paper falls short in terms of how the method is evaluated. I feel like too many things are buried in the appendix, and it remains unclear to me if MIER can, under realistic circumstances, really adapt to OoD tasks. I agree with R2 that Meta-World might be a suitable benchmark to evaluate MIER on, since  the training and tests tasks are distinct and there's a clear evaluation protocol for ML10 and ML45 (you have to adapt within 10 episodes) which would make it easier to compare to existing methods. 

Given the above I stand by my initial rating.
review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a novel meta-RL algorithm that can both meta-train and adapt to new tasks in an off-policy way. Multiple ideas are incorporated into the algorithm. Meta-learning parameters for both the context variable and the policy then updating them through gradient descent endows the policy with strong learning ability, while the dynamics model allows the reuse of training data by relabeling them. The authors conduct experiments in two types of task settings, which are in-distribution tasks and out-of-distribution tasks. Experiments show that MIER outperforms some existing methods on out-of-distribution tasks in terms of sample efficiency, while achieving comparable results on in-distribution tasks. The work is original, and has a certain contribution to the meta-RL setting.

## Pros:
1. Adaptation ability by modeling the dynamics. Leveraging the strong ability of gradient based models and consistency of the dynamics model, MIER can adapt to both in-distribution and out-of-distribution tasks. 
2.Sample efficiency by reusing training data. Most gradient based methods require a large number of samples to calculate the loss function for gradient descent. Although a similar off-policy idea has been adopted in MQL, the authors use the training data in another way, which is by relabeling them. This technique weakens the assumption in MQL, thus is suitable for both in-distribution and OOD tasks.
3. Clarity of the paper. The authors clearly describe the motivation of the proposed algorithm, which is to achieve sample efficiency on OOD tasks. The authors extend the meta-RL problem setting to in-distribution and out-distribution tasks with varying reward functions and dynamics, and conduct experiments on these settings with multiple existing methods. The experimental results are sufficient to support most of the claims.  

## Cons:
1. Assumption of simple MDP. The authors assume that a few steps of exploration are enough to capture the key information of tasks and guide the dynamics model, which might be insufficient in more complex changes in the MDP.
2. Complexity of the algorithm. In the experiments, the authors treat the information whether the task is in-distribution or out-of-distribution as known, which is unrealistic in most cases. The full adaptation process, which requires two stages of policy gradients and one stage of data relabeling, is much more complex than that of most existing methods.
3. Limitation of the dynamics model on OOD tasks. The adaptation to OOD tasks is based on the assumption that the dynamics model gives true or near-true transitions, which limits MIER’s ability. In the experiments, the authors conduct experiments that do not change some properties of MDP, more specifically, the state transition remains the same given the desired action. For some other meta-tasks where the fundamental of MDP changes, e.g. Walker-Rand-Params, the dynamics model may output undesired state-action pairs on the unseen tasks since it was not trained on these MDPs.
4. Uncertainty of MIER. Fig. 5 shows a strong uncertainty of the training curves and the training process does not seem to have converged for all experiments but Half-Cheetah Velocity. Since the paper focuses on adaptation to OOD tasks, the authors should discuss this phenomenon in the main body of the paper.
5. Necessity of data relabeling. Fig. 6 raises a question for the necessity of data relabeling in terms of the final performance, since both MIER and MEIR-wR converge to similar performance in most OOD cases.

## Questions:
1. Assume the information whether the task is in-distribution or out-of-distribution is unknown, so that data relabeling is performed anyway, will MIER still achieve a good performance on the in-distribution tasks?

Although the paper is well-organized and mostly well-written, I would like to point out some typos:
1. On page 5, in Algorithm 2: N_m is not mentioned in the algorithm.
2. In section “Experimental Evaluation” on page 6 & In C.1, Appendix C: “mujoco” -> “MuJoCo” .
3. In section “Citation”: The authors should pay more attention to the rules of references.
review_title: This paper proposes a convincing algorithm to adapt to both in-distribution and out-of-distribution tasks, while the expressiveness of the dynamics model limits the adaptation ability of the algorithm.

Meta-review: The paper was evaluated by 4 knowledgeable reviewers and got mixed scores. While most reviewers appreciated the new intuitive approach to meta RL. there were severe concerns about algorithmic choices and the evaluations that led to a poor score from some reviewers. These concerns are summarized below:
- The motivation of experience relabeling for out of distribution samples is not clear (R2)
- It is unclear why experience relabeling does not work for in distribution samples (R2, R4)
- The reported performance is not a fair comparison as it is typically not known when a task is in-distribution or out-distribution, so we would either have to take always experience relabeling or never (or learn when do use which algorithm)
- The paper falls short in terms of evaluations (R3, R4), in particular it remains unclear to me if MIER can, under realistic circumstances. It is suggested to use more established benchmarks such as Meta-World to evaluate the performance of MIER. 

For the given reasons, I recommend that the authors do these corrections and  go through another round of reviews at another conference.

Decision: Reject