Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Targeting offline meta-reinforcement learning, the work proposes a model-based method called MerPO, with conservative value evaluation and individual policy improvement method with the tradeoff between the meta-policy and the behavior policy influence.

The main algorithm MerPO includes an initialization step and a two-loops meta-learning approach. The initialization step learns the model of the meta-model and dynamics for each task. With the fixed estimations of the models of tasks, the proposed merPO will alternatively update the policies for each task and the meta-policy.

The main contribution claimed in this paper is that the proposed method is a more robust method with the design of a regularization term involving the behavior policy, which improves the meta-learning policy when the behavior policy is actually better than the current meta-policy. Theoretical guarantees are displayed and experiments are conducted to show the performance of MerPO.
main_review: Advantage:
The writing of the paper is clear and easy to follow. The proposed MerPO outperforms the state-of-art baselines and is intuitively explained well. The experiments show most of the performance of the components in the proposed algorithms, model-based vs model-free, involving behavior policy or not, the influence of the alpha, etc.

Disadvantage/Problems under concern:
1. About the important regularization parameter $\alpha$. 1) Will an adaptive $\alpha_t$ be better? Since the quality of the meta-policy $\pi_c$ will be different during the learning process. 2) Do we need to use different alpha for each task when the behavior policy property of each task is different? This scenario seems didn't been discussed enough?
2. How to choose alpha without information on the performance of the behavior policy and meta-policy? Do we need to try it out?
3. There are some undefined or unclear notations, such as what is $\rho_n$ in equation (5). 



summary_of_the_review: The main contribution of this paper is well claimed and verified. With a small revision, the algorithm can achieve better performance on the examined tasks in this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a model-based meta-RL approach to offline learning over a distribution of MDPs (offline meta-RL). The proposed method is inspired by COMBO and in fact is a direct extension of it to the multi-task setting. The extension involves meta-learning a model-initialisation (or prior) that can be rapidly adapted to new offline RL tasks. They further add a proximal RL policy-improvement operator, where the prior policy is meta-learned over the task distribution. They establish theoretical guarantees for their method and conduct a careful empirical investigation to motivate its design, while establishing that it provides additional benefits compared to relevant baselines. 
main_review: This paper is motivated by what the authors refer to as an inherent trade-off between exploiting the data in the offline dataset, and exploring out-of-distribution states (by means of a model). Empirically, they motivate this trade-off by noting that the quality of the data-set has a large impact on the efficacy of offline meta-RL approaches, and that single-task approaches to offline RL can be superior if data-quality is high.

To allow out-of-distribution exploration, they meta-learn an initialisation for a state-transition model via standard supervised meta-learning approaches. This initialisation is then used to rapidly adapt the model to a given dataset. The main part of the paper revolves around how to meta-learn a policy that can act as a regularizer in a policy-improvement step. They demonstrate empirically that if task-adaptation is regularised only towards the meta-learned policy, then performance can be surprisingly poor in the even that the meta-policy does not correspond to useful behaviour on the new task. Instead, the authors propose to regularize policy-improvement updates both towards the meta-learned policy and towards the behaviour policy implicitly defined in the offline dataset.

While the authors make a good case for their design choices, my main issue with this paper is that it complicates the presentation unnecessarily, which obfuscates connections to prior works. As far as I can tell, the latter regularizer (towards the behaviour policy) corresponds to COMBO, but this is not made clear until several pages later in the experimental section. Hence, the gist of this paper is to extend COMBO to the multi-task setting by meta-learning, while also meta-learning a prior policy for proximal policy updates. This could be made much clearer, which would not only help place the contribution in proper context but also make the paper much more readable.

A main strength of this paper is the careful motivation of each component of the proposed method. The authors demonstrate theoretically that both these regularisers are important for policy improvement, clearly motivating the need for a meta-learned prior as well as regularisation towards the dataset's behaviour policy. This observation holds also empirically, as the authors demonstrate that their method is at least as good as using only one of the two regularisers under various assumptions of data-quality. They also demonstrate that learning a model is critical for performance, and that their proposed method is better or on par with established offline meta-RL baselines.

Overall, while I think the presentation of the method can be made much simpler and clearer, I believe this paper presents interesting findings for offline RL and has a strong proposal for an offline meta-RL algorithm.
summary_of_the_review: I recommend acceptance of this paper. Main strengths of the paper are:

+ Compelling motivation of algorithm
+ Careful analysis of its design
+ Competitive performance

Main weaknesses are
- Overly complicated presentation of the algorithm

Post rebuttal: 

I've read other reviews and the author's rebuttal and maintain my recommendation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors argue that offline meta-RL methods do not learn from poor data well, by demonstrating that COMBO (a single-task offline RL method) outperforms FOCAL (an offline meta-RL method) only when the training data is good. They use this to motivate a method called MerBO, which involves (1) learning a meta-dynamics model with proximal meta-RL and (2) updating a policy with real and synthetic data using a method called “RAC”, which is equivalent to COMBO but with an added KL regularizer against the behavior and meta-policy, and (3) updating the meta-policy used by RAC. The authors prove that under certain assumptions, the resulting policy will outperform both the meta-policy and the behavior policies. The authors compare the method to existing offline meta-RL methods and demonstrate that (1) the use of a dynamics model is important, (2) the addition of the behavior cloning regularizer is important, and (3) the use of the dynamics model is important to perform well on held-out offline RL tasks.
main_review: This paper studies an interesting question of offline meta RL and presents promising experimental results. The ablations do a good job of demonstrating that each component of the method contributes to the success of the overall method. 

One concern with the paper is that the motivation for the method is rather unclear. This lack of clarity is due in part because the paper does not make specific, falsifiable claims. For example, one question posed is, “Why does the proximal Meta-RL method in Eq. (5) perform poorly in offline Meta-RL, even with conservative policy evaluation?” The paper continues to say that, “A poor meta-policy may have negative impact on the performance” and that, “following the meta-policy may lead to worse performance.” It is always possible that something “may” lead to worse performance, and this does not directly motivate the conclusion that, “it is necessary to balance the tradeoff between exploring with the meta-policy and exploiting the offline dataset, in order to guarantee the performance improvement of new offline tasks.” It would be great for the authors to provide direct evidence that this trade-off is necessary.

Similarly, the results in Figure 1 do not justify the claim that, “Clearly, existing offline Meta-RL fails to generalize equally well over datasets with varied quality.” Figure 1 only looks at one specific offline meta-RL algorithm and on one environment. Making such a general claim would require much more extensive evidence, and I do not see how, even if the claim were true, that would directly motivate the need to develop a method that will “strike the right balance between exploring with the meta-policy and exploiting the offline dataset.”

In short, the paper does not provide evidence that striking *this* balance is the main issue that must be addressed.

Another concern with the paper is that it is a relatively complex combination of existing components (COMBO, proximal meta RL, behavior cloning regularization), and so the onus is on the paper to demonstrate exceedingly good results to justify the complexity of the method. Although the results are positive, the method only mildly improves over FOCAL, and the paper would be strengthened by comparing to MACAW and BOReL, as those methods have been shown to perform well in these environments.

Another concern is that it is unclear how alpha is tuned or can be tuned in practice. The authors state that, “It is worth noting that different tasks can have different values of α to capture the heterogeneity of dataset qualities across tasks” but in practice, choosing a separate α for each task seems undesirable. If α was tuned for this method but no hyperparameters were tuned for baselines, this would also be concerning as this would bias the results in favor of MerPO.

There are some unjustified, or at least confusing claims, such as:

“Because tasks are trained on offline datasets, value overestimation (Fujimoto et al., 2019) inevitably occurs in offline Meta-RL” Overestimation, as far as I know, is only an issue with value-based methods.

“we study a more general offline Meta-RL problem.” I do not see how the problem statement in this paper is more general than the offline meta-RL problem present in past papers.

“Learnt dynamics models not only serve as a natural remedy for task structure inference in offline Meta-RL, but also facilitate better exploration of out-of-distribution state-actions by generating synthetic rollouts” Evidence that learnt dynamics model help specifically because they facility better exploration is not provided.

“Our results also provide a guidance for the algorithm design in terms of how to appropriately select the weights in the interpolation” I do not see how the theoretical results can practically guide appropriate choosing alpha. In the end, it seems like alpha still had to be chosen empirically.

Lastly, I have a clarification question: Can the method be applied to new, non-offline meta-RL tasks? One limitation of the approach is that RAC seems to depend on having a behavior policy to regularize against, making it impossible to use the resulting policy for online meta-RL.
summary_of_the_review: The paper does not provide compelling evidence that their method addressed a critical problem, and the experiments make it difficult to know if the method proposed really outperforms current meta-offline RL method since details on tuning and important comparisons were not included. Moreover, the overall method is rather complex and potentially limited to the pure-offline RL (unless I am mistaken) evaluation scenario, making the practicality of the method questionable.

Edit: Based on the rebuttal, I've increased my score up to a 6.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers the challenge in offline meta-RL problems, particularly, the difficulty with balancing a learned policy between exploring OOD state-actions by following the meta-policy and exploiting the offline dataset. To solve the problem, it starts with proximal meta-RL approach in the online setting, points out the problem of a degraded meta-policy in the offline setting due to lack of feedback from online interactions and proposes to regularize the task policy with additional penalty from deviating from the behaviour policy. It also integrates the model-based offline RL method for single tasks (COMBO) and offline meta-learning for task specific dynamics into its final form (MerPO). This paper provides theoretical analysis on its advantage over behaviour policy and the meta policy. Extensive experiments with ablation show the empirical performance meets its expected behaviour.
main_review: This paper proposes a new model-based offline meta-rl algorithm under the actor-critic approach. It is a nice addition to existing algorithms for the study of meta-RL problem. The difficulty of balancing the exploration with meta-policy and exploitation with offline dataset is an interesting observation, and the proposed solution is a reasonable remedy.

The main idea and the corresponding algorithm are well presented. The paper starts with a clear motivation in the introduction. After explaining the background, it points out the limitation of applying online meta-RL approach to the offline setting in section 4.2.1, and proposes its solution to introduce another regularisation around behaviour policy. The full algorithm include many components, meta-learning for dynamics, model based policy optimisation, conservative Q-learning, regularisation with behaviour and meta-policy. I am glad that the authors explain each component clearly and how they interact with each other.

The solution to add a second regularisation to task-specific policy seems simple and intuitive, but it is nice to show the proposed method does improve the performance both theoretically and empirically. The main theory results look reasonable but I haven't checked the detailed derivation in the appendix.
The experiments results are convincing and the ablation study is very helpful to show the contribution of each component.

I have a few questions about the algorithm design:

- CQL already encourages the Q function, and consequently the learned policy, to stay close to the behaviour policy. Why do you think it's not enough and we need another explicit regularisation to pull the policy close to the behaviour policy?

- Instead of regularising task-policy to behaviour policy in Eq 7, we can also consider regularising the meta-policy to behaviour policy in order to prevent the quality of the meta-policy to decrease. Could the authors comment on this option?

- Selecting a proper alpha value is the key to the proposed algorithm. Unfortunately the range provided by Theorem 1 is not computable and it seems hard to find an appropriate value according to the available offline dataset of each new task in practice. While the authors show 0.4 is a good value for all the experiments, I doubt if we can trust it in general. Hoping a constant value of alpha to work for all is counterintuitive because the whole paper is arguing that we should balance the two regularisation according to the quality of the dataset and the meta-policy. So we should adjust the interpolation coefficient accordingly.








summary_of_the_review: The paper has a good motivation and provides a reasonable solution to the problem in offline meta-RL. Theoretical and empirical results support the advantage of the proposed algorithm over recent baseline algorithms.
review_title: null

Meta-review: **Summary**

This paper proposes a novel offline model-based meta-RL approach called MerPO. MerPO combines conservative value iteration with proximal RL policy iteration.  The proposed method is novel despite having some similarities to approaches like COMBO. The paper compares against it in the experiments. The paper provides both empirical and theoretical justification for the proposed approach.

**Final Thoughts**

Overall, I think the authors did a pretty good job at addressing the reviewers' concerns. Overall, I think this is an interesting contribution to the ICLR community. The reviewers were all positive about this paper. For the camera-ready version of the paper, I would recommend the authors to go over the reviewers' concerns again and make sure that those concerns are addressed in the paper too as they did in the rebuttal. Some captions are pretty short; for example, see the captions of figure 6 and figure 7. I would recommend the authors add more description to the captions in the camera-ready version of this paper.
Decision: Accept (Poster)