Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tackles offline meta-RL where new tasks are learned via online few-shot adaptation. They observe that there is a distribution shift between the offline data used for meta-training, and the data collected online by the meta-learned policy to use for adaptation. This distribution shift can result in poor adaptation performance. To remedy this, the paper proposes to adapt the policy using only the trajectories with the highest returns - using return as a proxy for out-of-distribution detection. The method is evaluated on the MetaWorld Benchmark where it outperforms prior approaches. 

Contributions
an offline meta-RL algorithm that can adapt online to new tasks within a few trajectories, without knowledge of the reward function or offline data for the task.
Define the distribution shift between behavior policy and learned policy and show that OOD context data leads to poor adaptation
Empirical evaluation demonstrating that the proposed algorithm outperforms prior approaches on a standard meta-RL benchmark
strength_and_weaknesses: Questions / Comments
1) If I understand correctly, the proposed method is to use the returns of few-shot trajectories collected at test time as a proxy for measuring whether the collected data is in distribution. Then adaptation is performed using only the highest-return trajectories as context. In Section 4.2, the motivation for this proxy is stated in one sentence, referencing prior works Fujimoto et al. 2019 and Kostrikov et al. 2021. However, this proxy seems like quite a blunt instrument, conflating areas of the state-action space with low return in a particular task with areas outside the support of the offline data distribution. Consider the didactic example in Section 5.1 - the agent cannot make use of trajectories that don’t reach the goal to update its task belief, because those trajectories are deemed OOD because they have low return? I think I am mis-understanding the method, could the authors please clarify? If I am understanding correctly, then the method is quite limited. 

2) It would strengthen the paper to show a baseline that tries to detect OOD data in a more standard fashion, e.g., estimating uncertainty via an ensemble.

3) In the point robot experiment in Figure 3, if the training data is all clustered around a trajectory that goes straight (light blue bubbles), how is it that the exploratory trajectories sampled online navigate coherently outside of this distribution (green dots)? As an aesthetic nitpick, 3a is quite small and hard to parse.
Nitpick in wording: On page 1, “BORel (Dorfman et al., 2021) and SMAC (Pong et al., 2022) employ few-shot online adaptation, but the former assumes known reward functions, and the latter requires free interactions with the environment without reward supervision.” It seems that the advantage of the proposed method over SMAC is that thousands of samples (albeit unsupervised) from the test environment are not required. This sentence doesn’t really say that, and makes it sound as if collecting data without reward supervision is onerous, which is not the case. 

4) In Section 6, how is the distribution shift discussed in this paper different from the distribution shift discussed in Pong et al. and Dorfman et al? I’m not as familiar with Dorfman et al, but in the case of Pong et al. it seems that this paper is tackling exactly the same issue as raised in that paper. It’s not a problem to tackle the same problem as other papers, and can lead to confusion not state so clearly.

clarity,_quality,_novelty_and_reproducibility: Clarity - Parts of the paper are clearly written; however Sections 2 and 4 are difficult to parse. I remain unsure if I understand the proposed method correctly. 
Quality - theoretical and experimental results are thorough
Novelty - The proposed way to address the offline to online distribution shift in offline meta-RL is novel, though the components exist in prior work. However, unless I am understanding the method wrong, it is very limited in that it cannot acquire information from low-return trajectories in order to adapt to a new task. 
Reproducibility - It is not clear if code will be released publicly. The appendix contains a table of hyper parameters which will aid reproduction but will likely not be sufficient given the complexity of meta-RL implementations.

summary_of_the_review: The paper tackles a relevant and important problem in the field of meta-RL -- addressing the distribution shift present when meta-training offline and learning new tasks online. The experimental results show significant improvements over prior work. However, my current understanding of the method is that it is not able to distinguish between OOD trajectories and trajectories that achieve low return in a given test task, and thus can only adapt from high-return trajectories, severely limiting the applicability of the method. Therefore I don't think it currently meets the bar for a contribution for this conference.

--------- Update 12/6/22 ----------

Thank you to the authors for your detailed response, which has clarified a misunderstanding I had about the method. There is an assumption that the offline meta-training dataset is collected by expert policies, and thus only contains high-return trajectories. In my first review, I missed this assumption, which is stated in Section 2.2. I would recommend that the authors make this assumption more explicit in other parts of the paper as well. For example, the introduction refers to “task-dependent” policies, but it is not clear that this implies expert policies. 

With this clarification, I understand that part of the distribution shift being addressed is between high-return trajectories during meta-training and potentially low-return trajectories collected during meta-testing. Figure 3 is now clear, thank you!

However, the proposed approach of handling this distribution shift by filtering out low-return trajectories has great limitations that in my view diminish the contribution significantly. The method cannot make use of low-return exploration trajectories to update the task belief, so exploration during meta-testing is constrained to sampling the unadapted meta-learned policy. Thus the method effectively does not address the problem of exploration in meta-RL, which is a key challenge. While the experimental results do show improvement over prior methods (the prior methods are not designed for online adaptation and so are severely handicapped, but I am not aware of an existing work that is), I think the methodological limitations are severe enough to outweigh the empirical contribution. 

I think clarity of presentation can be further improved; specifically, to make the problem statement very clear. I also find the notation in the theory presentation to be difficult to follow. Limitations of the method should be clearly discussed in the main body of the paper. 

Overall, I think that the methodological limitations outweigh the contributions, and that the paper would make a much stronger contribution if this limitation is addressed. Therefore, I maintain my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work considers the problem in offline meta-reinforcement learning, where there is a distribution shift between the contexts seen in the offline data, and the contexts obtained by online roll-outs. This work first offers some formal characterizations of this shift, and then uses these characterizations to motivate a new offline meta-RL algorithm that works by filtering out episodes that appear out-of-distribution in order to obtain contexts that still look in-distribution for online adaptation.
strength_and_weaknesses: ## Strengths

There is much to like about this work. This work clearly and soundly characterizes the problem that it is trying to solve (with some exceptions, described below), and overall, the main idea of filtering out trajectories that the policy doesn't know how to update on from online contexts is a novel and interesting approach to offline meta-RL. Additionally, this approach appears to yield fairly impressive empirical improvements that are relevant to the meta-RL community.

## Weaknesses

The theory could be strengthened in a few areas:
- Specifically, the notion of filtering introduced in Fact 1 is insufficiently precise and requires some work to formalize. One precise notion of filtering is to filter based on visiting belief states under the transformed BAMDP that are also visited by the behavior policies (i.e., are present in the offline data). However, this still requires some work to formalize: how do you deal with multi-step episodes, where the first few timesteps include belief states that are out-of-distribution from the offline data, but later transition into in-distribution beliefs? Intuitively, inclusion of such episodes seems beneficial for the post-adaptation policy, but it requires more precise filtering definitions.
- It should also be noted that the existing notion of filtering based on visiting transformed belief states is generally infeasible. For example, in the case where different behavior policies take distinct actions at the initial state, but the observation after taking these actions does not reveal information about the identity of the current MDP, the transformed belief update will erroneously place all mass on the MDP corresponding to the observed behavior policy, which cannot be corrected without actually knowing the identity of the MDP. At the very least, it would be worthwhile to discuss under what cases such filtering is actually possible.

This work would also benefit from discussion about its own limitations and acknowledgements of which aspects are heuristics or approximations. Specifically, the idea to practically filter in GCC based on the trajectories earning the highest reward is heuristic that does not work in general. In many tasks requiring "challenging" exploration, such as those considered in HyperX [1] or DREAM [2], highly informative exploration trajectories do not necessarily achieve high returns, and would be filtered away in GCC. Other times, it may be impossible or difficult to obtain any high reward trajectories, even when these trajectories are highly informative, which would also cause failure. This work would be strengthened from discussion on this aspect. Additionally, it may be interesting to discuss relationships with the original theoretical motivation of matching behavior policy trajectories or more precisely, testing for in-distribution-ness with the behavior policies.

[1] Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning. Zintgraf et al., '21.

[2] Decoupling Exploration and Exploitation in Meta-Reinforcement Learning without Sacrifices. Liu et al' 21.

## Additional Comments
- There is an error in Proposition 1, where the policy evaluation gap is in fact $\frac{H^+ - 1}{2}$, and not $\frac{H^+}{2}$.
- It's unclear why the example in Figure 2 includes $2v$ actions, where the last $v$ actions yield no reward -- this appears to have no bearing whatsoever to the provided theory, and could probably be omitted for clarity.
clarity,_quality,_novelty_and_reproducibility: Generally, the clarity of this work is quite high, except in the theoretical sections highlighted above.
The quality and novelty is good: the idea to filter out trajectories is new and quite interesting, though it would benefit from discussion on when such a method would not work, as discussed above.
summary_of_the_review: Overall, the ideas in this work are quite interesting and are useful to the community. However, as there are some significant weaknesses to address, I will begin by only weakly recommending acceptance, though I am willing to increase my score based on the authors' response.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper identifies the distribution shift between the offline dataset and online rollouts as the core problem for offline meta-RL and adopts a Bayesian inference procedure to handle it.
strength_and_weaknesses: Strength
- Empirical performance is strong compared to the baselines.
- The proposed method is intuitive.

Weakness
- The authors introduces the term "data distribution mismatch" and describes it as a challenge unique to offline meta-RL. However, I am confused with this term in two ways: 1) "data distribution mismatch" seems too broad a term to indicate a problem specific for offline meta-RL. 2) It is unclear why this problem is unique to offline meta-RL. Distribution shift is a problem that persists for almost any learning models. In what aspect is data distribution match different from distribution shift? In other words, why do we need this new term?
clarity,_quality,_novelty_and_reproducibility: Mentioned above.
summary_of_the_review: The paper proposes a data distribution correction method to efficiently 'guess' a probable task during policy deployment. While the clarity of the paper can be improved, the proposed method is intuitive and shows better empirical performance compared to the baselines.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a greedy context-based meta-testing procedure to tackle the distribution shift issue when performing online adaptation of offline meta-trained policies. The proposed method extends FOCAL with a diverse latent task embedding sampling strategy and a context selection mechanism based on the highest return in the meta-testing phase. The authors justify such a methods with analysis in the BAMDP framework. The experiment results in several standard offline meta RL benchmark datasets demonstrate the superiority of the proposed method over previous offline meta-RL methods in the online adaptation setting. 
strength_and_weaknesses: ### Strength

1. The proposed method is well-justified by the theoretical analysis and the proposed method performs well compared to baseline methods. 

### Weaknesses

1. The clarity needs to be improved (see my detailed comments below).
2. There’s no discussion on the limitations of the proposed method. Specifically, the proposed method still assumes that the meta-testing task is sampled from the same task distribution as the meta-training tasks. Thus, the method is dealing with data distribution shift between offline data and online adaptation data, but not distribution shift between meta-training tasks and the meta-testing task (e.g., in cheetal-vel when all meta-training tasks correspond to velocity targets < 1 m/s while the meta-testing target velocity is 2 m/s). I encourage the authors to make it more clear in the paper.
clarity,_quality,_novelty_and_reproducibility: ### Clarity

1. Could the authors provide a more detailed discussion on the advantages of the proposed method over SMAC and BoRel? Why aren’t these two methods included as the baselines? 
2. The main idea of this paper, i.e., purely relying on online adaptation experience to identify the task instead of relying on the expert-level offline data, should be made much more clear early in the paper. I was confused in terms of the difference between this work and FOCAL/MACAW after reading the introduction. 
3. Figure 3(a) is confusing. The offline dataset for the point robot environment should consists of multiple goals (meta-training tasks) but all offline data correspond to the same goal. It exaggerates the distribution shift between offline data and online adaptation trajectory. 
4. Similarly, I couldn’t get the idea of Figure 1. Please provide an explanation. Why is the meta-testing task out-of-distribution? Isn’t it identical to task 3? 
5. The diverse sampling process described by equation (7) assumes that closer latent task embeddings indicate closer policies. Does this mean that the proposed method may only work with *some* off-the-shelf offline meta RL methods that explicitly force this kind of latent task embedding structure, e.g., FOCAL with contrastive loss? 

### Quality

This is a solid paper with extensive theoretical analysis and experimental results. However, I find that a few parts of the paper need to be improved in terms of the clarity. 

### Novelty

The greedy context-based meta-testing is novel in the offline meta-RL setting. 

### Reproducibility

The authors provided the code.
summary_of_the_review: This is a borderline paper and I’m willing to adjust my score if my concerns outlined in my above comments (mostly regarding clarity) could be addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper considers an offline meta-RL setting with few-shot online adaptation where in the test task no data is given ahead of time and the agent relies on its own exploration to learn the new task. It provides theoretical analysis on the data distribution shift that is caused by the different behavior policy, that is, task-dependent behavior policies in training tasks versus offline meta-policy in test tasks. It proposes to filter out-of-distribution data during online exploration and use filtered data to update the posterior of test tasks. It then introduces a greedy context-based data distribution correction approach, GCC, and filter data according to returns. Experiments on Meta-World ML1 benchmark shows SOTA performance compared to recent offline meta-RL baselines.

Strengths:
- It studies an important problem in offline meta-RL
- The theoretical analysis of the data distribution shift caused by different behavior policies in the offline meta-RL setting is novel and insightful. This is different from various distribution shift previously studied, e.g., in meta-training/test tasks in meta-RL setting and the offline-online policy shift in single-task RL setting.
- The proposed filtering based approach to mitigate distribution shift is theoretically justified.
- SOTA performance in the offline meta-RL online adaptation setting that none of prior works studied.

Weaknesses:
- The particular instantiation of the proposed filtering approach that uses return to filter data has a severe limitation. As pointed out by multiple reviewers, it will ignore low return trajectories that could be used to update the task posterior distribution. Also, filtering out low return trajectory will cause the agent fail to learn in a task that requires exploration. The method would not apply to offline dataset collected by low quality policies either. This limitation is not sufficiently discussed or demonstrated in the experiments.
- Clarity could be further improved. Multiple reviewers express confusion about the particular OOD problem the paper studied, the behavior policy in meta-training. While the rebuttal provide much better clarification than the submitted version, the paper could be further improved with more thorough explanation.
justification_for_why_not_higher_score: The limitation of the algorithm may lead to severe drop of performance but is not well discussed even after rebuttal.
Decision: Reject