Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents Model-based Adaptable Policy LEearning (MAPLE), a model-based offline RL algorithm that learns a policy that can quickly adapt to an unknown environment during execution. Existing offline RL algorithms constrain the policy to visit only in-support regions, but MAPLE aims to model all possible dynamics models for out-of-support regions and try to learn a dynamic-conditioned (context-conditioned) policy. Policies for various uncertain dynamics models are prepared first, and gradually filter out policies that do not correspond well to transition observations when deployed. However, solving it exactly is intractable, and a practical approximation is presented, which uses an ensemble dynamics model set and an RNN-based environment-context extractor. Experimental results show that MAPLE outperforms baselines in D4RL benchmarks, showing better generalization ability.

main_review: Overall, the paper is well written and easy to follow. In contrast to existing offline RL algorithms whose computed policy is fixed when deployed, MAPLE can exploit new experiences obtained during test time and can exhibit fast adaptation to the true environment. I found this research direction interesting and novel in the context of offline RL.
- Still, I have some concerns when compared with meta RL, sim2real, domain randomization techniques. It would be great to provide some explanations of what makes MAPLE distinct from those approaches (e.g. applying meta RL whose task distribution was simply replaced with the uniform distribution over the ensemble of the learned dynamics model).
- In the experiments, MAPLE is the only algorithm that uses RNN-based policy and exploits new samples obtained during deployment. Thus, it might be unsurprising that MAPLE outperforms the baseline algorithms. Then, for example, can VariBAD [1] (where the task distribution is defined by the ensemble of learned dynamics models) be comparable with MAPLE? If so, it would be great to see the comparison results.
- Why the reward penalty $-\lambda U(s,a)$ is required for MAPLE? What will be the result without such uncertainty penalty?
- In Figure 2, if the transition after taking $\beta$ is totally unknown, we need to consider infinitely many 'stochastic' transition models, rather than considering deterministic transitions only.
- line 101: $p(\tau | \pi)$ -> $p(\tau | \pi, \rho)$

[1] Zintgraf et al., VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning, 2019

limitations_and_societal_impact: Limitations have been discussed in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Conservatism, i.e. avoiding out-of-distribution actions (and states), is a major theme in offline RL because of the need to avoid erroneous extrapolation beyond the data. This paper takes an alternative approach and instead trains a policy that adapts when deployed beyond the data distribution. To do this, a context encoder RNN is trained to produce latent codes given the episode history, and the encoder and policy are jointly optimized to maximize average performance across a large ensemble of pretained dynamics models. The algorithm, offline Model-based Adaptable Policy LEarning (MAPLE), is shown to achieve strong results on the MuJoCo tasks from the D4RL benchmark. 
main_review: The algorithm is largely a combination of the ideas of MOPO and PEARL, but the combination is novel to my knowledge and well-motivated in the setting discussed. It is different than PEARL in that the meta-learning is happening in a model-based fashion across different dynamics models. However in that sense it is conceptually related to MB-MPO (https://arxiv.org/pdf/1809.05214.pdf), an algorithm which meta-learns a policy via MAML to quickly adapt to any model in the ensemble. I think MB-MPO should be cited and discussed in related work.

The theorem appears correct, and perhaps useful for high-level intuition regarding when/why the algorithm succeeds, but the result is not very informative in my opinion because all the heavy lifting is being done by the unknown coefficient $N_m$, for which no further analysis is presented.

The results on the D4RL MuJoCo tasks look good! One suggestion you may consider to further improve the paper is to experiment with tasks that require leaving the behavior distribution in order to succeed. This was one of the motivations of MOPO, and the authors developed two tasks specifically for this purpose (cheetah-jump and ant-angle).

The writing of the paper is clear and I have no significant issues with it.
limitations_and_societal_impact: They discuss the limitations thoroughly (which is appreciated), but not the societal impacts.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose a new offline model-based RL method MAPLE. The method relies on training an ensemble of dynamics with the offline data in addition to an environment-context extractor model that maps from dynamics to a latent embedding, which is part of the input to the policy. The authors claim such an environment-context extractor could both eliminate inconsistent dynamics and thus the remaining policy, after the probing stage, is the optimal one w.r.t. the ground truth dynamics during deployment. The experiments show that MAPLE achieves competitive results compared to state-of-the-art offline RL algorithms. 
main_review: Strengths:

1. The development of the paper is well-established. Sec. 4.1 provides a good warm-start in the ideal situation, and the theory result gives good intuition on the tradeoffs and algorithm designs. Though there are still gaps between the transitions that could be fixed (please see below).

2. The algorithm is clearly presented, along with the network architecture provided in the appendix. Seems easy to reproduce the results. 

3. The empirical results seem promising. The algorithm achieves competitive results compared to the most state-of-the-art offline reinforcement learning algorithms. 

Weakness:

1. The definition of the optimal environment-context extractor $\phi^{\ast}$ (in line 194) seems quite strong. The current definition imposes a strong assumption that there exists a single policy that is optimal for all dynamics models. The assumption could be more  reasonable if it is $\exists \phi^{\ast}, \forall \hat{\rho} \in \mathcal{T}, \exists \pi_{\phi^{\ast}} \in \Pi, J_{\hat{\rho}}(\pi_{\phi^{\ast}}) = max_{\pi}  J_{\hat{\rho}}(\pi)$. This is also because the paper seems to treat the policy and the environment-context extractor as separate classes, and it seems more realistic to assume a global optimal $\phi$ but not policy $\pi$ that is optimal for every dynamics model (as Eq.1 suggests, the best we can do is to seek for a policy with maximum expected returns on the distribution of the dynamics models).

2. The introduction to the set of dynamics $\mathcal{T}$ seems abrupt. At the first glance, it could be hard to follow why one suddenly needs to optimize the policy over a set of dynamics since the results in the previous section assume a convergence to a single policy/dynamics at the end. It could be better if an explanation is provided here for the dynamics model set (e.g., introduce stochasticity while all models are deterministic, or model ensemble, etc.).

3. The reviewer is not fully convinced of the "probing" ability of the environment-context extractor. Across the algorithm there is actually no dynamics-related signal while training $\phi$. As described in the paper, "If the optimal actions in the same state are conflicting, to increase the performance of objective, Equation (1), the policy gradient has to backpropagate from $\pi$ to $\phi$" (line 223-224). Since there is no guarantee that the learned dynamics could extrapolate accurately in out-of-support regime, training $\phi$ purely based on the returns could also make $\phi$ exploit the errors of the models. For example, even if the dynamics of the deployment environment is included in the dynamics set. If all dynamics are consistent up to timestep $i$ but there exists a dynamics that induces a larger expected return than deployment dynamics then the policy, together with $\phi$, will more likely select the optimal action of that particular environment, other than the optimal action of the deployment environment. 

Originality: 

This work is an incremental work that improves on current model-based offline reinforcement learning algorithms that combine ensemble model learning and pessimism. 

Clarity/Quality: 

Most of the paper is well written. Some of the technical details could be better explained (as suggested in the weakness section).

Significance: 

The major contribution of the work is the empirical competitiveness of the proposed algorithm. The reviewer is not fully convinced by the technical contribution of the work (namely, the technical soundness of the environment-context extractor model).

limitations_and_societal_impact: This work does not seem to have a negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to learn adaptable policies in an offline RL setting. One of the limitations of offline RL methods is decision-making in the out-of-support region. This paper tries to leverage existing meta-learning algorithms to learn a latent code z from trajectories as a context variable. Then, a context-conditioned policy would take in both states and context variables for generating actions. They experimentally show this method outperform previous methods on standard offline RL benchmarks. 
main_review: The paper considers a novel setting wherein the deployment time, the learned policy might need to adapt to have the best performance. They borrow the techniques from meta-learning to learn context-aware policies that can adapt.  The paper is well written for clearly explaining the motivation, intuition, and practical considerations. The paper would contribute toward realistic offline RL methods.
limitations_and_societal_impact: Mujoco locomotion tasks are standard for offline RL, but in D4RL, there are many more meaningful data that can be used to test the proposed method. It usually remain unclear what happened in the environment while using locomotion tasks as a testbed. For example, except for seeing improved scores, it is unclear to me what has been improved what has been not. Is it really out-of-support? Is context-variable z informative? Can you visualize z? 

MAPLE-200 only helps in 2/3 of the tasks. This seems a bit contradictory to the theory part.
review_title: null

Meta-review: Most reviewers agreed that the paper has an interesting new way of performing offline rl (i.e., how to do well in out-of-support regions), and the empirical results on standard benchmark are very promising. 
Decision: Accept (Poster)