Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a reward prediction module to produce reward labels in a meta-RL setting. Here the offline RL part requires reward labels while the online portion can be done in a self-supervised manner by generating reward labels using the offline trained module.

main_review: Strengths
+ It is a simple addition to the meta-RL setup and is shown to alleviate the z-distribution shift.
+ Related work is well covered.

Weakness/Comments
- Some of the exposition is a bit high-level. Grounding in running examples would be helpful to better understand the overall approach. For example, in a set of tasks how different can any two tasks, is the difference at the level of one being picking and one being opening a drawer or is the difference at the level of different pick locations in a pick task.
- '... can autonomously interact with the environment ...' is a very strong assumption for many real world tasks. This needs automatic resets in place as well as safe policies that already allow the agent to explore (which if are already available and robust defeats the purpose of learning to some extent?).
- The claim that '... reward prediction generalization is easier than policy state, action, and z generalization' is not sufficiently supported. If the reward decoder is not well trained or reliable then wouldn't the bad labels skew the learning in the wrong direction? It wasn't clear how this issue was being addressed or if this imposes limits to the overall approach and where it can be used.
summary_of_the_review: The idea is simple and works on the problems tested. The larger applicability and constraints of the approach are not clear. Explanation of the approach can be improved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to enable effective meta-RL in the offline setting. It first identifies the problem of simply combining meta-RL with offline RL: distribution shift in z-space. This paper then proposes a hybrid method that uses offline datasets to meta-train an adaptive policy and then collects and utilizes additional unsupervised online data to bridge the identified distribution shift. Empirical results show that unsupervised online learning significantly improves the adaptive capabilities of the meta-trained policy.
main_review: ## Merits

1.	The problem this paper aims to address is practically important for RL, that is, how to effectively utilize offline datasets to meta-train policies and enable them for fast adaptation to new tasks. 
2.	This paper identifies a specific problem to offline Meta-RL : distribution shift of the exploration policy for collecting data for adaption, which looks interesting.
3.	The idea of using unsupervised online data collection seems novel and shows some effectiveness in alleviating the problem of the distribution shift during the adaptation. 
4.	This paper is generally well-written and easy to follow. 


 ## Concerns and limitations
1.	The technical contribution of the proposed approach is limited. It is basically a simple combination of AWAC and PEARL. One modification to this combination is to use unsupervised online data collection, whose data are labeled with synthetic rewards, instead of ground-true rewards. However, synthetic reward learning is not novel and looks straightforward.
2.	Another concern is that the proposed approach seems to assume tasks share the same dynamic model and only differ in reward functions. This is because the proposed SMAC method learns the task encoder only through the backpropagation from the reward learning, in contrast to prior work like PEARL. This assumption may bring SMAC with some advantages over other more general methods in tasks only differing with rewards, but will significantly limit its applicability. How will SMAC perform in settings where tasks may have different dynamics?
3.	The reviewer is also curious about how this approach works in sparse-reward tasks? 
4.	From the experiments, the outperformance of SMAC is mainly due to the unsupervised online data collection, which may not be a fair comparison to baselines. Without this online data collection, SMAC underperforms MACAW in two out of three tasks. 
5.	This paper only evaluates the proposed method in three scenarios. It is highly recommended to include more settings to strengthen the empirical results.
6.	In Algorithm 1, Line 8 is confusing. What is $D_i$ in this line?


summary_of_the_review: This paper identifies a specific problem of distribution drift in offline Meta-RL and proposes an interesting idea of using unsupervised data collection to alleviate this problem. This paper is generally well-written. However, the proposed approach has limited technical contributions and makes strong implicit assumptions. The experimental evaluation also needs to be improved by including more settings.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Meta-RL methods aim to learn to learn using a set of meta-training tasks. Yet, it requires a large number of online interactions for training. Offline meta-RL methods aim to learn from static datasets labeled with rewards for different tasks. These methods usually suffer from distributional shifts between the behavior policy from the offline data and the meta-test time exploration policy. This paper aims to tackle this distributional shift issue of offline meta-RL methods. To this end, the paper proposes a framework that is robust to the distributional shift by introducing a self-supervised learning phase. In this phase, the proposed framework leverages online data without reward labels and generates synthetic reward labels for it based on the labeled offline data, allowing for smoother adaptation. The experiments show the proposed approach alleviates the distribution shift issue and thus marginally improves post-adaptation results compared to offline meta-RL prior works. I believe this work tackles an interesting and promising research direction (i.e. alleviate the distributional shift issue in offline meta-RL) and proposes a convincing framework to address it. The experimental results verify the effectiveness of the proposed framework. Therefore, I am leaning toward accepting this paper with some concerns on the assumptions made by this work and its applicability.
main_review: ## Paper strengths and contributions
**Motivation and intuition**
- Tackling distributional shifts of the exploration policy in offline meta-RL is well motivated.
- Introducing the online self-supervised learning phase is intuitive and convincing.

**Novelty**
- I believe the introduced online self-supervised learning phase is novel.
- The proposed self-supervised learning method is an intuitive way to learn from the online self-supervised learning phase. 

**Clarity**
- Writing is easy to understand. Approach, notations, and figures are well explained.
- Figure 2 well supports the targeting problem and motivation.
- Figure 6 qualitatively showed targeting problem is resolved.

**Related work**
- Authors clearly address the additional assumption (online phase) from related work and its motivation.

**Experimental results**
- The presentation of the experimental results is clear.
- The experimental results demonstrate that the proposed approach 
Figure 5 shows that self-supervised learning gradually improves meta-agent even without reward labels.

**Reproducibility**
- The code is provided, which helps understand the details of the proposed framework.
- Appendix gives essential details for implementation. I believe reproducing the results is possible. 

## Paper weaknesses and questions

**Offline datasets**
This work (and offline meta-RL prior work) assume the availability of offline/static datasets labeled with rewards for different tasks, which could be very expensive to obtain in some cases, limiting the applicability of this line of work. This work uses scripted policies to obtain the offline datasets, which means that all the tasks are solvable already. I would like to hear the authors' opinions on how this can scale up to more complex domains.

**Suboptimal data**
The robot succeeds on the task in 46% of the transitions in the offline dataset, which is a pretty low success rate. I wonder if the proposed framework works better partially because of the suboptimality of this dataset.

**Online interactions**
The proposed method needs additional online interactions compared to offline meta-RL prior works. While it could be cheaper to collect, sometimes it can still be impossible to leverage online interactions.

**Reward prediction performance**
It would be informative to see how well it can predict the rewards of online interactions and how this affects learning performance.

## Other metrics

### Relevance and significance
Solid contributions to a relevant problem

### Novelty
Worthy contributions, but not surprising

### Technical quality
Technically adequate for its area, solid results

### Experimental evaluation
Solid, informative evaluation w.r.t all 5 criteria

### Clarity
Very clear, only minor flaws.
summary_of_the_review: I believe this work tackles an interesting and promising research direction (i.e. alleviate the distributional shift issue in offline meta-RL) and proposes a convincing framework to address it. The experimental results verify the effectiveness of the proposed framework. Therefore, I am leaning toward accepting this paper with some concerns on the assumptions made by this work and its applicability.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work investigates the meta offline reinforcement learning problems. It highlights an issue called "distribution shift in z space" in the meta offline RL. In the meta-test phase, the agent needs to explore the test environments and the collected transitions are used to generate z representing the task feature. The policy is conditioned on the task feature z, so it is important that the transitions collected by the exploration policy are informative enough to extract the feature information. In the offline RL setting, the distritbution of transitions collected by the exploration policy deviates from the distribution of transitions in the static training dataset. Therefore, inferring task feature z becomes problematic.

This paper proposes an additional online training process (without the reward info) to fix the issue of "distribution shift in z space". It learns a reward function from the offline training dataset, and thus it is able to generate the reward for transitions collected during online training. These online training transitions with the estimated rewards are used to update the learned policy.
main_review: Strengths:

This paper notes a problem called "distribution shift in z space" in offline meta RL and illustrates this issue in Figure 2.
This paper is well written and easy to understand.
The visualization of the distribution shift at the end of the experiment part is informative. 

Weaknesses:

The problem of "distribution shift in z space" might require deeper discussion. In other context-based meta RL algorithms (e.g. RL^2, MQL, CaDM), where the task context feature is extracted from a sequence of past historical transitions, is there still the problem of  "distribution shift in z space". If yes, could the proposed unsupervised online training be beneficial to solve this issue of "distribution shift in z space"  in these algorithms?

In the online training process, is the distribution of tasks the same as tasks in the offline training dataset? Figure 1(right) seems a bit confusing. The task set T1, T2, T3, T4, T5, T6 in online training is different from the task set T1, T2, T3 in the offline set. Then when we compare with offline meta-RL (Figure 1 left) or online meta-RL (Figure 1 middle), the proposed method is exposed to more tasks during training? Is it a fair comparison?

In section 5.2, in the online training process, if the agent interacts with the environment of task T_i to get a trajectory \tau, does the proposed method use D_i (offline dataset collected on the task T_i) to infer z and then generate reward? Or it is fine to use D_j (j is not equal to i)? How accurate is the estimated reward in comparison with the ground truth reward? Does the advantage of the proposed method heavily rely on the accuracy of the estimated reward? If yes, under the sparse-reward tasks where estimating reward is challenging (e.g. the sparse 2D navigation task in PEARL), will the proposed method fail in this case?

Is it possible to demonstrate the advantage of the proposed method on standard offline meta-RL benchmarks (e.g. D4RL)? The results on standard benchmarks with commonly used offline datasets will be more convincing.

summary_of_the_review: The online training process with generated rewards needs more clarification. And it will be better if the proposed method is evaluated on standard offline RL benchmarks.
review_title: null

Meta-review: The paper describes a new offline meta RL technique that addresses the distributional shift problem with a self-supervised online exploration phase where reward labels are not available.  The framework is novel and interesting.  The authors addressed many concerns of the reviewers.  However, the additional experiments raised additional questions.  For instance, why does meta-BC perform so well, even better than the proposed method without online data, and other baselines seem not to work at all?  In the discussion, the reviewers expressed concerns about the experimental results in the case of changing dynamics.  Those experiments are questionable since the proposed method only considers the reward information to deal with different dynamics.  Finally, an important question regarding SMAC remains unanswered: how much does the proposed method depend on the quality of the offline dataset and the quality of the reward decoder? Overall, the work is promising and the authors are encouraged to continue their work by addressing the reviewers concerns.
Decision: Reject