Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper is doing quite a few things at once:

1. Proposing an algorithm for learning effective exploration policies for meta-RL from only offline data
2. Defining the offline meta-RL problem (not technically new but concurrent with other recent work)
3. Defining the MDP ambiguity problem, which proves troublesome for inference-based meta-RL algorithms, and providing two baseline approaches to solving it

The authors accomplish 1 by converting the VariBAD to work with off-policy data and 3 with intuitive strategies for additional data collection or augmentation, which provably address the problem. The experiments suggest that the proposed solutions are necessary to solve problems for which identifying the MDP is difficult.
main_review: Overall, the paper is generally well written and focuses on an interesting, important problem. The solutions presented are intuitive and well-motivated, and the experimental results match the theoretical claims. My primary concerns are (a) the impracticality of the data collection/augmentation solution to MDP ambiguity and (b) the explanation of the ambiguity problem and how identifiability is connected to overlapping identifying states.

### Writing

The writing is clear. A concise and informative recapitulation of necessary technical background information needed to understand the problem is given; the Background section has pedagogical value.

One confusion I had re: L149: since R^+ and P^+ are unknown (the original data comes from regular MDPs), isn't it somewhat unsurprising that the VariBAD objective doesn't refer to these explicitly?

### Novelty & Significance

The paper describes an off-policy variant of an existing algorithm, VariBAD, as well as two data collection/augmentation approaches to solving the MDP ambiguity problem. While the solutions are not particularly surprising, but they are well-motivated and explained clearly. Overall, the contributions are sufficiently novel and their effectiveness suggests they are significant as well.

### Experimental Evaluations

The experimental evaluations are generally intended to assess whether the proposed algorithm learns useful exploration policies as well as whether or not MDP ambiguity presents a problem in practice. The experiments are reasonably convincing relative to other papers in this area, although several environments are very simple.

### Remaining Questions

Overall, I think this is a solid paper that probably deserves acceptance on the basis of clear motivation, well-motivated solutions, and reasonably convincing experiments. Again, my biggest point of confusion is regarding the connection of identifiability of data (used in the theory) to the (not clearly defined as far as I can tell) notion of "overlapping states." I also have some concerns that the proposed data relabeling/augmentation approaches are not viable in practice. In order to raise my score, the authors could convincingly answer a few questions:

- Can the authors more clearly define what "overlap" (e.g. L214) means and how it is connected to identifiability? Right now, there is a gap in my understanding of how the identifiability is linked to the specific experimental domains presented, because the identifying states are defined in terms of *pairs* of MDPs, not single MDPs.
- Although the authors mention that constraining the hypothesis space of MDPs is not desirable, this approach would be attractive because it does not require additional data collection/augmentation. Can the authors comment on the viability of such an approach to solving MDP ambiguity problem, or conjecture what such an approach might look like?
- Is extrapolation error not a problem when applying off-policy VariBAD to fully offline data, as is a problem in offline RL?

*Update after discussion period*: in light of the authors' responses to my questions as well as the other reviewers, I have decided to raise my score to 7. I think the characterization of the MDP ambiguity problem is a meaningful contribution that will be of interest to the NeurIPS community, even if the proposed solutions to it are relatively straightforward. The authors can improve their paper (as discussed previously in this forum) by further explaining the significance of the problem they describe and the practical limitations of the solutions that they propose.
limitations_and_societal_impact: Some discussion of safety during exploration for RL/meta-RL systems would be welcome. In addition, the authors do not describe the technical limitations of their work. In particular, describing settings for which the proposed data collection/augmentation strategies are not practical would be worthwhile.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper studies the problem of Offline Meta-Reinforcement Learning (OMRL). In this setting, one observes M trajectories coming from N different environments sampled from the same underlying distribution. The goal of the agent is to perform well in expectation on a new unseen MDP drawn from the same distribution. To do so, the authors propose relabelling strategies to mitigate the problem of MDP ambiguity they introduce/formalize and to enable off-policy learning for the VariBad algorithm. The resulting algorithm was tested and compared with PEARL (online) and MBML (offline) on a series of environments with discrete or continuous action spaces, and sparse or dense rewards.
main_review: **Pros**

- The problem investigated in this work is novel and will certainly gain importance in the future. The paper poses the problem well and qualitatively discusses its challenges, lying the groundwork (MDP ambiguity) for future contributions.
- The relabelling strategies seem simple to implement and efficient in mitigating the MDP ambiguity introduced in the paper.
- I liked the different environments used, with both sparse and dense rewards highlighting the difference in performance when using the relabelling strategies.

**Cons/Questions**

My main concern regards the positioning of the paper as doing offline RL. The setup certainly is but the solution isn't.

- The policy relabelling assumes that one can access the learned policies and that additional data can be collected. The offline RL setup would be greatly simplified if one could make such an assumption (in practice). I acknowledge that the authors clearly stated this issue -  L235-237: "We importantly note that changing the data collection method in hindsight is not suitable for the offline setting. Therefore, the proposed scheme should be viewed as *a guideline for effective OMRL data collection*." Nonetheless, the paper is still advertised as doing offline RL.
- The reward relabelling suffers the same limitation. The authors assume that the reward of transitions can be relabelled. This requires access to the reward signal. I agree this is a softer assumption as opposed to having access to the full environment dynamic. The authors mentioned this is not a problem - L254-257: "We remark that the assumption on known reward (during training) is mild, as the reward is the practitioner’s method of specifying the task goal, which is typically known; this assumption is also satisfied in all meta-RL studies to date." For meta-RL it might be true, but this is a hard assumption when working in an offline setting.

**Reason For Rating**

I consider this paper as a good stepping stone on which the research community can build. The framing of the problem, as well as the discussion around the ambiguity in identifying the MDP, is well conducted. The experimentation is fair and adequate to highlight the benefit of their approach. However, I would encourage the authors to re-position the paper as the offline setting is violated on multiple occasions in their solution. Despite these, I consider the work to be a valuable contribution to the research community and I could be swayed if my concerns are answered.
limitations_and_societal_impact: There is no social impact to consider in this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses the problem of learning Bayes-Optimal Exploration strategies from offline data.
The authors discuss MDP ambiguity problems that may arise in this context and propose BoRel an adaptation
of the online, on-policy VariBad algorithm that addresses these issues via policy replaying and reward relabeling.
Finally, the algorithm is empirically assessed on several baselines and compared to a number of existing (online) methods.
main_review: The paper studies a highly relevant problem setting and is technically sound. It is very clearly structured and well-written. The discussion of MDP ambiguity is novel and highlights an important avenue for future research. However, several remaining weaknesses are listed in the following:

1. The adaptation of VariBad for off-policy learning seems to be of limited novelty. The authors claim that VariBad makes no use of the BAMDP parameters. However, the VAE component (approximately) learns to compute a sufficient statistic of the interaction history which the policy network is conditioned on. This can be interpreted as a posterior over the BAMDP parameters and thus the VariBad policy network acts on hyperstates (which is known to be enough to enable computation of the optimal policy).
Switching out the on-policy component with an off-policy one seems trivial as does the re-computation of belief posteriors once the inference VAE is ``  `  fully'' trained.

2. On the topic of training the VAE in the offline setting. The trajectory distribution induced by the final policy is likely to differ significantly from the one induced during data collection. Unless I am mistaken, this would be equivalent to changing the data distribution between training and test time in a ``standard'' supervised context - something that can lead to rather arbitrary behaviour. In light of this, I would be curious as to how a reasonably robust belief computation can be ensured?

3. I find the proposed policy replay strategy somewhat problematic in the context of the offline meta-RL setting. The authors even note that data generally might not be collected specifically for the meta-RL task. Consequently, I do not belief that the agent should be able to prescribe data collection policies.

4. The proposed reward relabeling assumes access to the reward functions of all example tasks which is somewhat limiting. Additionally, it seems to be strongly related to other data augmentation techniques e.g. Hindsight Experience Replay which should at least be mentioned, if not discussed in more detail. 

5. While the issue of MDP ambiguity is clearly described and well-illustrated through examples, its consequences in practice remain somewhat unclear. Are there any results regarding the best possible performance one can hope to achieve in its presence? What happens when we cannot make the additional assumptions required for policy replay and reward relabeling? 

6. The authors compare to online PEARL claiming it to be a strong baseline, however, other meta-RL algorithm have achieved higher performances (e.g. VariBad itself). Additionally, since PEARL is an online algorithm it needs to both curate its own data (which in BoRel is already done by the expert policies, policy replay and reward relabeling) as well as extract the optimal meta-policy from that data. This implies that PEARL is solving a harder problem and the comparison is somewhat unfair. 
limitations_and_societal_impact: -
review_title: null

Meta-review: This paper introduces an interesting problem, where the meta-agent is required to quickly learn unseen tasks given some offline data for training tasks. In addition, the paper also formally introduces the MDP ambiguity problem induced by the new problem setup. Although the proposed solution (off-policy version of VariBAD + reward relabelling) is a little bit straightforward, it is a reasonable solution to the problem considered in this paper, and the empirical results are also solid. During the post-rebuttal discussion period, the majority of the reviewers agreed that the formal description of the MDP ambiguity problem and the Bayesian RL perspective is interesting enough to be presented at NeurIPS. Therefore, I recommend accepting this paper.
Decision: Accept (Poster)