Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a method for "fully" offline meta-RL. Specifically, they assume there is no interaction with the environment at all neither during meta-train nor meta-test and this method only sees previously collected data at all times. Their method is built on top of AWR [1] in which policy updates are weighted by the advantage term.  Except for section 4.3 and the last paragraph of section 4.1 (see below), the paper is well-written and easy to follow.

My comments:
- While this paper touches on a very interesting and practical problem in meta-rl and batch-rl, I didn't find their setting is very realistic with respect to batch and offline RL setup. In batch RL, even though we assume there is a fixed data for training ( no interaction with an environment whatsoever), the final performance will be measured by interaction with the environment (i.e. policy will be evaluated in an online setting). However, this paper assumes that there is no such interaction with the environment exists. The reason the original setup of batch-rl is important because it imposes lots of challenges such as extrapolation error, over-estimation, etc which are very critical in real-worlds and make batch-lr an important problem. However, in this paper, this is not the case. The question is how the setup in this paper is sensible and important at all?

- I am not sure about the "consistent" definition in this work and it doesn't make sense to me. Per paper definition, if an algorithm can find a good solution to test tasks regardless of the meta-training task distribution, is called consistent. My understanding from this definition is training tasks are not important at all and test-tasks can have a very different distribution from training data and still work. If that is true, why we need training tasks in the first place? based on this definition, can't we just randomly initialize the algorithm and do a meta-test and get the same results? (it would be a good experiment to do as well)

- For the experiment section, this paper uses the benchmark introduced by [2, 3] and it compares with PEARL. Did you follow PEARL setup for the experiments or ProMP/MAML?  PEARL samples a fixed set of tasks at the beginning of training while ProMP and MAML samples a set of tasks at every iteration. It is important because PEARL works in the former setups not the latter. In addition, you used the same parameters as PEARL for the experiments in this paper which is not fair as experiments (i.e. fixed dataset) in this paper are different from PEARL. PEARL might get better result with hyper-parameter tuning. 

- I am not convinced that why one should select L_\pi in the inner loop instead of L_{AWR}? why not just use L_\pi in both the inner and outer loop? Can you explain the motivation for this choice? Section 4.1 didn't make the case for this selection.

- Experiments are done for only 1M steps. Why 1M for all algorithms as they are different? it is like supervised learning,  you can train for more gradient setups and it is likely results will change ( with hyper-paramaters tuning).  

- Looking at Figure 6, MACAW doesn't show a good performance vs. others. Why this is the case? is there anything special about this experiment that MACAW doesn't work? 

- What is the difference between R(s,a) in eq.1 and R(s) eq 2? shouldn't  R(s) in eq 2 be R(s,a)?

minor: Figure 7 and 8 don't have legends. 

[1] Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019 

[2] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks

[3] Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search.

review_title: My review 

final_score: rating: 5: Marginally below acceptance threshold
review_content: - Summary:
    - This paper makes two contributions:
        - 1. Formalizing the offline meta-RL paradigm, where we meta-train on pre-collected (offline) data for several RL tasks and adapt to a new task with a small amount of data. Within offline meta-RL the experiments focus on the fully offline case, where the meta-test task is also offline. It could be online with the meta-train tasks remaining offline, but then we would have to meta-learn an exploration policy, which isn't done in this work.
        - 2. Introducing MACAW: an algorithm for offline meta-RL that has the desirable property of being consistent (i.e. converges to a good policy if enough time and data for the meta-test task are given, regardless of meta-training). To do so they rely on MAML (which provides consistency) and AWR (a simple, popular offline RL algorithm) and add a couple of changes: some hyper-network like parameterization to add capacity and adding an extra objective in the policy update to enrich the inner loop.

- Pros:
    - 1. The paper proposes an important area of research
    - 2. Most of the experiments are well executed, using good baselines and as well as providing understanding through ablations
    - 3. MACAW is a nice simple algorithm with good guarantees.

- Cons:
    1. I think the offline meta-RL paradigm is not introduced correctly. 
        - In particular, the paper largely borrows the meta-RL formulation from the online setting where task=MDP. It then treats the collection of the batch data as an obvious after-thought once the task is defined. However, in the offline setting the policy that generated the batch of data is of critical importance and should be part of the task definition. 
        - For instance, it is not the same to receive an MDP and examples from the perfect policy (where you can take supervised examples of a given s) than from a random policy (where there is no signal) or an adversarial policy that tries to act as bad as possible. This also has consequences at the meta-level: if at meta-training time I only see examples of perfect policies I may learn to imitate them, then at meta-test time I see an in-distribution MDP but data coming a bad policy and imitating it is a bad idea. In the definition of 'task' described in the paper this is fine since task=MDP. However, we would expect this to work poorly since the policy is out-of-distribution with those since at meta-training.
        - This effect of the quality of the data is used in the experiment of figure 4 left, which makes the experiment good (a Pro), but does not detract from having (IMO) the wrong formalization.
        - This is a big minus since this formalization is one of the big contributions of the paper and it could affect further papers in that area. However, there is a chance I am wrong since the authors have spent months on this and I've spent only some hours reviewing the paper.

    2. One of the two improvements over MAML+AWR, the weight transform, is not fully justified:
        - From a conceptual point of view, it's not clear to me why this is the first time MAML has needed this change after being used in tens/hundreds of experiments. What is different on this task that hasn't been true in any other task in the past? I understood we're doing it to increase the representation capabilities of the gradient, but wouldn't this be useful for other meta-learning tasks? If so, why wasn't it used in the past? (specially since the bias-only version of the idea was already proposed by Finn et al. in 2017)
        - From an experimental point of view I couldn't find the details of the ablated version on the main text appendix B, D or E. Therefore, I understood we're simply changing it to just a weight matrix of the same dimension. If that's the case one could argue that to make things comparable we should try increasing the width by a factor of $\sqrt{c}$ (c defined as in the appendix) to have roughly the same number of parameters, as well as possibly the depth, since that would be a simpler change with a similar latency to the weight transform version.

    3. [edited post-rebuttal to correct inaccuracy] The fact that there is a recent/concurrent paper is not ideal. However, I didn't weight it in my consideration.
- Clarity: pretty high
- Significance: somewhat high, except for similar concurrent NeurIPS work
- Questions:
    - Figure 4 left I didn't understand if the quality changed at meta-test only or both at meta-train and meta-test.
    - Any conjecture on why PEARL performance goes down in cheetah-velocity?
    - Doesn't AWR rely on the policy providing the data being somewhat good already? Otherwise the advantage function may be very different for the policy that generated the data vs. the optimal policy.
- Details:
    - Figure 3 has logarithmic x axis and figure 4 (left, center) have linear x axis. It may be better to keep them the same.
    - "An important property of a meta-RL algorithms is thus its robustness" --> _offline_ meta-RL?
    - In Related work Kirsch 2020b,a seems to be the same paper cited twice
- Summary of review: because I believe the formalization has a big flaw and this paper is mainly about the formalization, I have to recommend rejection. I also have major concerns regarding the weight transform and the experiments that were done to prove its usefulness. The paper is otherwise good, interesting, well-written, and timely; I'm looking forward to the discussion and updating my review if my initial assessment was wrong.

============
Update after discussion with authors

I had two main concerns:
- The modification to MAML was unconvincing to me.
- The offline meta-RL formulation should include behavior policy as part of the task definition.

After a very detailed response from the authors, I am now happy with the response and extra experiments w.r.t. the MAML modification, but I still have concerns about the formulation. In particular, reading the final version I still think the policy giving the behavior data is treated as an after-thought and is instead assumed constant across all tasks. For instance, IMO figure 1 should contain multiple examples of the same "RL task" that are different "offline RL tasks"; i.e. learning to swim using guidance from a 3-year-old and learning to swim using guidance from Michael Phelps. This is one of the key differences, IMO, between meta-RL and offline meta-RL and given that this paper's main contribution is introducing offline meta-RL, I feel it really should be very clear about this point. It may be fine to first introduce the correct general version and then say something like "it may be useful to assume each RL task is given by an expert of roughly the same characteristics", i.e. we can assume behavior policy is constant across tasks. However, right now the original formulation directly borrows from regular meta-RL and I believe that may confuse future papers in offline meta-RL.

I've increased my score from 4 to 5 since I'm now less concerned about the MAML improvement, but I cannot recommend acceptance given my concern about the formulation.
review_title: Well-written paper on a nice topic, concerns about formalization and weight transform

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes the problem of fully offline meta-RL. Here, the idea is to leverage offline experience from multiple tasks to enable fast adaptation to new tasks. The paper distinguishes two settings of offline meta-RL, one where only the training data is collected offline and testing corresponds to sampling online trajectories, the other where both training and testing data are collected offline. The latter is termed as fully offline meta-RL and is the problem setting considered in this work.
The paper also proposes a method for the fully offline meta-RL problem based on the MAML method. They argue that a naive application of MAML with advantage weighted regression (a recently proposed approach to offline RL) is insufficient for this setting and propose to make the policy more expressive by including an advantage head that regresses the advantage conditioned on the state and action. The approach is evaluated on offline versions of 4 continuous control problems.

Strong Points
- The paper is well written and the problem setting is well explained.
- A solution is proposed for the fully offline meta-RL problem. The modifications to the policy functions are backed by theory and is also empirically verified to be helpful in the experiments.
- Extensive ablations on the various modifications to MAML+AWR confirm that the utility of the approach for the fully offline meta-RL problem.
- The authors also explore settings of good/bad adaptation data showing the robustness of the proposed method to quality of offline adaptation data as compared with MAML+AWR 

Weak Points
- While the offline meta-RL problem setting is well motivated, I am not certain that the “fully offline” setting is as interesting. Since the idea behind meta-RL methods is to adapt to new tasks quickly, it should be generally feasible to have a small number of online trajectories for adaptation.
- The benchmarks used for evaluation are really toy benchmarks. I would have liked to see performance on MetaWorld benchmark. Interestingly, this evaluation is present in the Appendix where the approach seems to perform worse. Moreover, one of the continuous control benchmarks (cheetah direction) in the main paper doesn’t have a held-out test and is more a proof of concept.

Overall, I find the ideas proposed quite interesting. But given my hesitation with the utility of the fully offline setting as well as the experiments, I am recommending a weak accept.

Questions for the authors:
- Can you elaborate on why PEARL doesn’t work at all on even the Cheetah-direction evaluation which doesn’t have any held-out test set?
- How many step of gradient descent are used in the inner loop for the MACAW method? How many steps of fine-tuning for the MT+FT baseline?
- Why is the MetaWorld evaluation not part of the main paper? Can you please comment on the worse performance of MACAW on this more realistic benchmark for meta-RL?
- It will also help if the authors can provide more motivation for the "fully" offline meta-RL setting.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a new problem setting in meta reinforcement learning, namely metaRL. Here the agent is trained on a fixed offline dataset, which distinguishes it from most metaRL algorithms that interact with the environment during meta training. The authors propose a gradient-based meta learning method (MACAW) to approach this problem, which uses an actor-critic method combined with advantage weighting which is an offline RL method. In experiments they show that this outperforms the offline metaRL method PEARL, and combining multi-task offline RL with AWR in a naive way.

Overall I like the paper and think the problem setting is very interesting, and I like the proposed method. I have to admit I was a little bit disappointed when I came to the experiments - they're fine, but not exciting. I also have a few concerns with the practicability of this method when it comes to real-world datasets, which is one of the main motivations for this work. For now I give a score of 5, but I'm open to increase this and look forward to the author's response! 

Pros:
- The paper is very well written, and easy to follow. Concepts are well explained and hypotheses clearly stated.
- The proposed problem setting is timely and relevant for the metaRL community.  I like how explicitly the problem setting is described and how it is related to existing work and problem settings.
- The proposed method MACAW is sound and all concepts and objectives explained well. Even though there are quite a few components, I think I would be able to re-implement the method just from the paper description.
- In the experiments on 4 MuJoCo benchmarks the proposed method outperforms two baselines, Offline PEARL and Offline MT+FT.

Cons: 
- I feel like the experiment section is not as strong as it could be. All the ingredients are there though and I think with some extra work the authors can easily improve their paper.
 - Throughout the paper, you stress the importance of the algorithm being consistent. However, I feel like there's an experiment missing that shows that MACAW behaves like a consistent algorithm. What I would like to see is, if there is a significant shift between training and task distribution, can MACAW recover from a bad initialisation? How long does it take, and how does this compare to say continuing to train PEARL? Figure 4 (right) slightly hints at this but given that MACAW is still quite good at 3 tasks, and with 3 tasks you can somewhat cover the range of velocities, I think this is not enough. I'm not even entirely convinced by this "consistency" argument (especially not with empirical evidence). The agent only has a small dataset available at test time, so why care about consistency? Consistency is *only* useful if I can actually train longer than that - but in this case, I can also just continue training whatever other metaRL algorithm I pre-trained (like PEARL, etc). So what does consistency buy me? 
 - The paper could also be considerably improved with a more realistic benchmark. A main motivation for the offline RL setting is that you can use real-world data and train a metaRL agent using this. So why not use something more realistic (even something like meta-world would already be cool)? Or maybe there exists an off-line dataset with real-world data where you don't actually execute the policy in the end (because you can't) but you can somehow compute how well it does (by comparing it with expert actions for example). 
- I wonder if considering the *fully* offline dataset makes the problem too easy. I can imagine that for many real-world applications where you want to deploy the agent on new tasks, it will have to gather the data to learn about the task by itself, instead of being given this in an offline fashion (since that might often not be possible!). The authors briefly discuss this at the end of the conclusion and cite concurrent work by Dorfman and Tamar (2020) who aim to do this. Would it be possible to extend MACAW to learn good exploration policies (doesn't even have to be online, there could be a separate exploration and exploitation phase)?

Other questions: 
- Are the authors concerned that the updates for the value function (Eq 2) suffers from high variance due to the Monte Carlo returns? In the MuJoCo tasks considered in the experiments this might not be such a problem due to (a) dense rewards and (b) "good" offline data. But could this become a problem if those two things are not given? How would you deal with this?
 - Given that the training data is pretty good since it comes from policies that are pre-trained (using "good data"), would it make sense to add a simpler, supervised imitation learning algorithm as a baseline? Since we're in the *fully* offline metaRL setting this should be possible?
- Would MACAW also work on the standard metaRL setting, where the agent is interacting with the environment during meta training and therefore responsible for collecting the data itself?

Side comments:
- Sec 5: I don't understand the second sentence; do you mean "on" this problem? Why is MACAW listed as a "sensible" approach? 
- Fig 4 left: The colours in the caption don't match the one in the figure.

-------------------------------------------------------------------------------------------------------------------------
UPDATE

I have read the other reviews and the author's response. 

Thank you for your thorough answer! It's great to see you've taken all feedback into account and updated the paper significantly. After looking through the changes in the paper I'm raising my score from 5 to 6. 

Some last comments:
- Several other reviewers also raised concerns that the "fully" offline setting might be unrealistic. I saw you added a motivation for this in Sec 3, which makes me a little bit more convinced. It would still be great to add an experiment where MACAW is adapted online at test time (entirely without offline data) like in C.2 but on in-distribution tasks.
- I understand how the reward function issues in ML45 could be the cause of the inconsistent results. Maybe ML1 would be a better choice at this point, or indeed waiting until v2 of the benchmark is released. 
review_title: review

Meta-review: The paper proposes a method for offline meta-RL, where we meta-train on pre-collected offline data for several RL tasks and adapt to a new task with a small amount of data. The paper assumes that there is no interaction with the environment either during meta-train or meta-test.  In this setting, motivated by the ide of leveraging offline experience from multiple tasks to enable fast adaptation to new tasks, the paper introduces MACAW, which combines the consistent MAML and the popular offline AWR, improving upon them by adding capacity through parameterization and adding an extra objective in the policy update. As a result, the MACAW proposed for the offline meta-RL has the desirable property of being consistent, i.e., converging to a good policy if enough time and data for the meta-test task are given, regardless of meta-training. 


Pros: 
+ Most of the experiments are well executed, using good baselines. Extensive ablations on the various modifications to MAML+AWR confirmed the utility of the approach for the fully offline meta-RL problem.
+ MACAW is a simple algorithm with theoretical guarantees; the modifications to the policy functions are backed by theory. 


Cons:
- The reviewers have concerns on the formulation of offline meta-RL. One major contribution of the paper is to introduce offline meta-RL. However the paper largely borrows the meta-RL formulation from the online setting where task=MDP. The reviewers think that directly borrowing from regular meta-RL as the formulation of offline meta-RL might be misleading. The reviewers suggest including behavior policy as part of the task definition for offline meta-RL formulation.

- Several reviewers raised concerns that the fully offline setting might be unrealistic. Although the author did add a motivation, the reviewers would be interested in seeing MACAW being adapted online at test time on in-distribution tasks. 

- Unfortunately, the authors accidentally revealed their names in one of the modified versions. 

Decision: Reject