Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### **Summary and Contributions of Paper** 
This paper modifies the work "When to Trust Your Model: Model-Based Policy Optimization" (Janner, 2019) to handle the meta-learning case. The definitions, paper structure, and algorithm flows are all very similar to (Janner, 2019). The main difference is that the model architecture is an RNN (and hence conceptually, a new hidden state vector h_{t} is introduced) to handle adaptation for different given tasks, whereas (Janner, 2019) used a probabilistic feed-forward network as the model.

Experiments are conducted on standard RL meta-learning benchmarks (e.g. ForwardBackward Halfcheetah), to compare sample efficiencies between different model-based techniques.

### **Strengths**
- Contributes to the literature in model-based techniques in the meta-learning setting, which is generally lacking. 
- Experimentation seems to show the desired result - i.e. the paper's method is more sample efficient than both PEARL and L2A.
- Paper shows that a small modification of (Janner, 2019) using an RNN can solve meta-learning problems in RL.

### **Weaknesses**
- A bit _too_ similar to (Janner, 2019). Novelty-wise, it seems like a small (but effective) modification of the work in (Janner, 2019) from model-based optimization in the single MDP setting to the meta-learning setting, by including a recurrent/RNN model. The main contribution seems to be this modification, but all other techniques of the paper (e.g. defining the correct terms such as a _branched rollout_, generating the concepts such as C(eps, eps'), etc.) are exactly the same in (Janner 2019). I think the paper overemphasizes the portions that have already been written in (Janner, 2019).

- A bit hard to read at times, due to the denseness of the notation. This is especially encountered in page 6, where there are large chunks of text + math without breathing room. I suggest that the authors provide a high level summary in the main body, especially because most terms have already been defined in (Janner, 2019).

### **Clarity Comments**
- Since the paper is very similar to (Janner, 2019), it may be useful to highlight the main differences (e.g. different coloring of changed lines in the Algorithmic boxes), rather than redefining everything already mentioned - it was difficult to parse which sections were novel/changed and which were from (Janner, 2019). 

Overall, the main benefit seems to be this RNN modification and relevant meta-learning experiments (which is why I give a marginal accept), but I do think that the authors can re-write the paper in a much simpler fashion.
review_title: Seems like the TL;DR is a modification of (Janner, 2019)'s method to use an RNN model for meta-learning.

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Paper summary

This paper focus on model-based RL on a POMDP setting (they call it "meta RL"), where the policy and model need to infer the current hidden state according to history. It provides a theoretical relation between true environment returns and the returns from learned models in a POMDP setting. And it also provides a practical algorithm called M3PO and shows this algorithm is more sample efficient than some meta-RL baselines in some continuous control tasks.

### Pros

- Extend an existing work ( Janner et al. (2019) ) to a POMDP setting
- Refine some theorems in Janner et al. (2019), by taking more important premises into consideration

### Cons

- The theory part of this paper is quite similar to Janner et al. (2019). It seems they just apply similar theorems to the POMDP setting. For example, theorem 1 in this paper is an immediate result when combining Theorem 4.1 in Janner et al. (2019) and Lemma 1 in Silver & Veness (2010). And the idea of using branched rollout to control model errors is not first proposed by this paper either.
- The proposed algorithm (M3PO) is also very similar to algorithm 2 in Janner et al. (2019), and they just use this algorithm in POMDPs. From an implementation perspective, there are only two differences: 1. using GRU as network architectures, 2. use PEARL in policy optimization. So I feel the novelty in the proposed algorithm is limited.
- In the experiments, it seems M3PO (their algorithm) only has significant advantages over PEARL on Halfcheetah-fwd-bwd and Ant-fwd-bwd. In the other environments, the performance gap seems not very clear. Also, in Fig 1, they only show the horizon of 0.2 M steps, which is a relatively short training time for these difficult tasks. And if we compare M3PO-long and PEARL-long, it seems M3PO-long is overall worse than PEARL-long.

### Questions

- Why MBPO ( Janner et al. (2019) ) does not support POMDP? Just because it does not use RNN?

### Minor points:

- The figures are pretty hard to read, especially fig 2.

### References

- Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Proc. NeurIPS, 2019.
- David Silver and Joel Veness. Monte-Carlo planning in large POMDPs. In Proc. NIPS, pp. 2164– 2172, 2010.
review_title: Limited novelty compared to the existing works

final_score: rating: 5: Marginally below acceptance threshold
review_content: 


## General Summary

The contributions of this paper are two-fold. For one, the authors propose a meta model-based RL algorithm working with a meta-policy and a meta-model. In addition, the authors provide a lower bound of the meta-policy performance.
The algorithm is compared against PEARL and L2A, there is a further comparison against M2PO in the appendix.

## Writeup

I find the writeup quite dense and difficult to follow. For one, the idea behind meta-learning is generalization across different tasks/environments. While this was introduced well in sections 3 and 4, the connection is missing in the experiment section and algorithm section. The notation can be better introduced.

## Pros

The theoretical analysis is the greatest asset of this paper, it is useful to have a bound on the actual performance of the policy that depends on the length of the model rollout, model error and policy distribution shift. Also, it is a valid extension of MBPO by Jenner et al. (2019). 

## Cons

The policy distribution shift and model error based bounds have been addressed in the model-based RL literature. What separates this paper is the extension to POMDPs and meta-learning formalism. The writeup is dense, although the idea behind is simple (branched model rollouts in POMDPs).

## Comments

Maybe it's because I don't work on meta-learning, but I think that you are really missing a clear description of the algorithm. Where are the task switches happening, how does the initial task distribution come into play? This should be clear from algorithm 2 in my opinion.

In Algorithm 2. you could introduce a loop over tasks maybe? I realize that in the environments that you used, task switches happen  during the episode, but I wouldn't rely on that in the algorithm description because it's a bit confusing, looking at the the algorithm itself, I don't see a meta-learning algorithm but rather a model-based RL algorithm for POMDPs. 

The result that performance degrades with increasing model rollout horizon is not surprising and is the consequence of error compounding. This is a well-known result coming from standard model-based RL and MPC literature.

Branched rollouts (although meta-branched as the authors name them) are also also not a novel addition, as the authors note in the paper. The improvement was introduced in earlier literature (Dyna, MBPO).

Couple of questions for the experiment section:

* Are the results for L2A correct? They seem to be too bad to be true based on the results reported in the original paper.
* In Fig. 1, by number of training samples it is meant number of time steps in the environment? 
* The variance of the performance is quite high for the ant fwd-bwd, much higher that the other methods. Any explanation for that?
* In Fig. 1, the legend should appear for the whole figure and not just for the upper-left graph.
* The performance of PEARL with long-run is better in 3/6 experiments and seems comparable in the others, does this mean that M3PO doesn't use new data efficiently? Also, I would be interested in knowing how the variance looks like for those longer runs.
* In Fig. 2, it is not really surprising that the performance degrades so much with the rollout length. What is surprising to me is that oftentimes the performance is best for k=1, indicating that there is not much use of the model except for 1 time step, this is in contrast to Janner et al. (2019) in the non-meta setting.
* I would be interested in seeing how the method generalizes across  OOD unseen tasks (not seen on train), such that the training and test distributions are different and then plot training vs test performance.

It's good that the authors make a clear comparison between this work and Jenner et al. (2019) because there is a lot of overlap and as I understand, code was reused. Nevertheless, I am concerned about the amount of novelty, 
this is an extension to POMDPs with a correction to the theory. 

Is the meta-learning formalism even needed for this kind of algorithm and evaluation? 

Text-intricacies:
* check that the references are correct, p. 10 you have a reference of the form:  MBPO. [link]
* sec. 3 par. 2 polity->policy



review_title: Theory improvement and small extension of existing work

final_score: rating: 5: Marginally below acceptance threshold
review_content: === Summary ===

The paper concerns model-based meta-RL. It exploits the fact that meta-RL can be formulated as POMDP in which the task indicator is part of the (unobserved) hidden state. Thus, the paper effectively analyzes and proposes model-based algorithms for POMDPs. The paper bounds the gap between the expected reward of a policy in the actual POMDP and the estimated model and then theoretically shows that this gap can be reduced when using dyna-style / branched rollouts instead of full rollouts under the learned model. Motivated by this finding, the paper proposes a Dyna-like algorithm for POMDPs. In the experimental evaluation, the paper compares its proposed method, M3PO, to two recent meta-RL approaches in a range of meta-RL environments for continuous control.

=== Merits of the paper ===

Overall, the paper contributes a new algorithm for model-based meta-RL which is neatly motivated by the paper’s theoretical analysis. Both the theoretical analysis and the proposed algorithm are sound. The experimental evaluations demonstrate that the algorithm performs similar or better than previous model-based meta-RL algorithms and thus could be a relevant contribution to the field of meta-RL.

=== Strenghts ===
- The experiments include relevant meta-learning environments and algorithms to compare to.
- The proposed algorithm is sound and motivated by theory.
- I have done a very simple simulation study, being able to confirm that the gap in Theorem 2 is indeed (much) smaller than the one Theorem 1.

=== Weaknesses / Concerns ===
- Both Theorem 1 and 2 seem to be a straightforward combination of the results in [1] and the fact that POMDPs can be cast as MDPs with history-states. Thus the theoretical contribution/novelty is quite limited.
- As expected, from my simulation study it seems that bounds are vacuous and their usefulness beyond motivating branched rollouts is questionable.
- The only difference to original dyna-like approaches is the fact that a recurrent model with internal/hidden state is used – thus the algorithmic contribution is small as well.
- Overall, the clarity of the paper could probably be improved a lot – Many paragraphs and sentences are hard read and lack vital explanations. For examples, see below.

=== Overall Assessment ===

In my opinion, the paper is a borderline case. Currently, I see the paper slightly below the acceptance threshold. Both the theoretical and the algorithmic contribution of the paper are small. Overall, the paper is a straightforward extension of [1] to POMDPs with meta-RL experiments. Yet, due to a lack of clarity, it is hard to read. Nonetheless, the proposed algorithm seems to be practically relevant for meta-RL - thus I am happy to hear other opinions and open to being convinced to increase my score.

=== Questions & tips for improvement ===

Appendix A.1 briefly discusses the differences to [1], and claims to derive similar Theorems under more appropriate assumptions / premises in Appendix A.7. How do the theorems in A.7. differ from the ones in Section 5? If the theorems build on more appropriate assumptions, why not use them in the main part of the paper?

Adding a simple simulation study that shows how much smaller the gap/discrepancy is, when branched instead of full rollouts are used, would require little effort and strengthen the claim that branched rollouts are preferable.

I found section 5.1. to be very confusing – It is stated that the gap can be “expressed as the function of two error quantities of the meta-model: generalization error due to sampling and distribution shift due to the updated meta-policy”. Indeed Def. 2 is probably the generalization error due to sampling. However, Def. 1 is the expected TV distance of the estimated model from the true model. Then, how should this be the “distribution shift due to the updated meta-policy”?

=== Minor comments ===

- Section 4, 3rd paragraph: It should probably be "$r_{t}$ and $o_{t+1}$ are assumed to be conditionally independent
- Section 5.1.: It would be good to have a proper definition if $\pi_{\mathcal{D}}$, i.e. the data-collection policy
- Theorem 1: The definition of $\epsilon_m$ is duplicate whereas the $\epsilon_\pi$ is not included
- Section 7, 4th paragraph: Should be “In Humanoid-direct, the performance”


[1] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. NeurIPS 2019

review_title: Fair paper - Straightforward extension of Janner et al. (2019) to POMDPs

Meta-review: The paper presents a meta-learning for Model-based RL that introduces branched rollouts to improve sample efficiency of the learned model.

While the paper addresses an important topic of sample efficiency in RL, and provides theoretical analysis, the reviewers raised concerns with the novelty and clarity. The extension to POMDP setting is certainly important technological contribution, albeit a straightforward. To be suitable for publication the work needs to make stronger case for the significance of the method.  
Decision: Reject