Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ##########################################################################

Summary:
 
The paper proposes an algorithm to improve on a 'semi-performant' policy on a real world system from logged data of this policy driving the system (offline setting).
The algorithm build a MPC based controller using as ingredients learnt models of the dynamics, reward and value function along with a clone of the semi performant policy.

The algorithm mixes in an original fashion the following ingredients
- soft reward weighted trajectory averaging to select the next action from sampled trajectories (PDDM).
- reuse of past MPC trajectories (linear mixture with next guiding samples)
- guided shooting with a cloned policy (POPLIN) 
- learning of ensembles to capture a notion of learnt model uncertainty (PETS)


##########################################################################

Reasons for score: 
 
The combination of the aforementioned ideas to the problem of offline policy improvement is original.
The paper is well written, the related work section is thorough the results  convincing.
 

##########################################################################
Remarks:
 
1. The model based approach to off-line learning is a very promising avenue. The explicit separate representation of reward and dynamics provides great flexibility as shown in the adaptation to new objectives or additional constraints.
 
2. The paper is close to PDDM as acknowledged by the authors in the introduction sections. 
The ablation study is interesting and shows that the behavior cloning is the key factor in the algorithm.
The reader is referred to Figure 2 and the results not discussed at all. 
I guess it is a space issue, but I'd like to read it. 

3. In the ablation section, the author say they recover PDDM when removing the policy and the value function.
I would suggest you give a more central place to this statement.
The RL literature is (incredibly) messy and discussing relation (inclusion, generalization, special case) between methods 
when available should be highlighted more.

#########################################################################

Some typos: 

(1) sec 2.1 'aim to to'
review_title: Good contribution to model based Offline RL. Well written, thorough evaluation

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper offered a model-based offline planning by taking advantage of learned behavior-cloning  policy and learned value function in addition to learning the model of the environment. Model learning is adapted from a work from Nagbandi et al. As the method takes advantage of behavior-cloning  policy, its performance boosts when the prior policy is reasonably well (medium or expert level). Moreover, learning value function helps the method to achieve higher rewards compared to the behavior-cloning  policy. Having said that, at least from the algorithmic point of view, this work does not seem to have a huge contribution. Moreover, it fails to provide a good solution compared to other model based learning methods when the behavior-cloning policy is not good (although it performs better than the BC policy). In fact, it looks like it would be a suitable method only when behavior-cloning  policy is in medium level (For the expert level vanilla BC gives comparable result). While this could have potential applications, the current work does not offer examples of it. If the authors agree to my conclusion about the medium level BC, finding (or even mentioning) the possible scenarios and environments could improve the impact of their work (And if they don't agree I am eager to know why). Moreover, it looks like due to their model-learning algorithm, the application of the method is mainly limited to robotics. 
Another point I want to raise is the conceptual similarity of this work with Value Iteration Networks (VINs). I think VINs are mainly for discrete state space tasks, but I am curious to know authors' opinion on this. Moreover, I think it is useful to mention it in the paper as a related work.      
Also, the authors mentioned that  "Along with this high-level design, many implementation details such as consistent ensemble sampling during rollouts, or averaging returns over ensemble heads, appear to be important for a stable controller from our  experience." which I believe is in favor of their potential contribution. However, in the conclusion they  said "MBOP is an easy to implement [...] algorithm". These two seem contradicting to me.   

Update: 
I thank the authors for their response and increase my score by 1. 
review_title: Have potential aplications, but needs improvement 

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary**

This work studies the offline RL problem and proposes MBOP for the same. The proposed method learns ensembles of dynamics models, behavioral policies, and value functions using the offline dataset. Subsequently, the approach uses online MPC with a learned terminal value function. The paper demonstrates experimental results on standard benchmark tasks (RLU and D4RL) as well as zero-shot adaptation results.

The strengths of the paper are showing results with an MPC-based approach and zero-shot adaptation results. My main concerns with the paper are exaggerated novelty, improper attribution and missing related works, and narrowly applicable approach. In the current form/version, I do not feel comfortable recommending acceptance. However, I am open to considering again if the authors can address the concerns during the rebuttal stage (which I expect will require a major revision).

----

(1) **Exaggerated novelty:** This paper must be considered in the context of a number of recent model-based offline RL papers. In particular, after MOReL and MOPO (which have been on arXiv for ~5 months before ICLR deadline), the use of model-based methods for offline RL, by itself, can no longer be considered a novelty. What sets this apart from MOReL and MOPO is the use of MPC vs a dyna style policy learning algorithm. This distinction and difference is not surprising or new, since a number of papers have explored this theme in the standard RL setting. However, these papers have not be adequately cited or discussed (e.g. guided policy search, POLO [1], AOP [2] etc). The MBRL tutorial (https://sites.google.com/view/mbrl-tutorial) has additional references too. I would recommend the authors rewrite the paper, to better situate the contributions of this work relative to what is known in literature. Right now, the first mentions of the most relevant prior work (MOReL and MOPO) appear only in page 5.

(2) **Missing related work:** Pointers to relevant works are missing. Notable omissions include Dyna based MBRL algorithms (e.g. Game-MBRL [3] and MBPO [4]). This also ties with the previous point about exaggerated claims of novelty. Furthermore, Game-MRBL also explores the adaptability of MBRL in detail. When the dynamics is the same but the reward function changes, then the approach considered in MBOP will work. However, when the dynamics is inconsistent or non-stationary (e.g. data for locomotion collected on the road but robot enters dirt on sidewalk), then the MBOP approach would fail rapidly.

The planning algorithm in PDDM (filtering + reward refinement) is based on the MPC algorithm used in POLO [1]. I would recommend the authors to cite both papers for proper scientific dissemination of information. Line 11 in algorithm 2 can be interpreted as a "learning rate" to shift a_t from current value towards a target value. This was proposed and analyzed in the work of Wagener et al. [4]. I recommend the authors to cite and discuss this work. The Adroit tasks were proposed in [5] but this paper is not cited.

Line 6 in Alg2 is equivalent to running a for loop over the models in the ensemble and an inner for loop running for N/K trajectories per model. Can the authors clarify if this is true? If so, the consistent trajectory sampling approach has been utilized in a number of prior works including Game-MBRL.

(3) **Narrowly applicable solution approach** : The paper relies on the use of a behavior cloning policy for regularization and assumes a single consistent policy is used to obtain the offline dataset. If multiple policies are used to construct the dataset, will the MBOP approach work? Do you propose to learn multi-modal generative models for the policies? Do you have experimental results in this regime?

----

**Update after author response**

I appreciate the authors efforts in the revision. This has addressed my concerns about the related work and (partly) exaggerated novelty. As a result I have increased the score from a 4 to 5. 

In my opinion, the distinction between MPC and policy optimization is minor -- policy optimization when given enough iterations can match an optimal planner, and similarly MPC when equipped with a good terminal value function can match a policy learner. The practical differences between MPC and policy learning have also been extensively explored in prior work (e.g. GPS, POLO, AOP etc). Thus, while MBOP provides an interesting case study in the use of MPC for offline RL, there is limited novel insight. Nevertheless, I appreciate the authors for the thoroughness of the case studies, and for updating the paper based on reviewer feedback.

---

**References**

[1] Lowrey et al. Plan Online Learn Offline. ICLR 2019.

[2] Lu et al. Adaptive Online Planning for Continual Lifelong Learning. 2019.

[3] Rajeswaran et al. A Game Theoretic Framework for Model Based Reinforcement Learning. ICML 2020.

[4] Janner et al. When to Trust Your Model: Model-Based Policy Optimization. NeurIPS 2019.

[5] Wagener et al. An Online Learning Approach to Model Predictive Control. RSS 2019.

[6] Rajeswaran et al. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. RSS 2018.
review_title: Review of MBOP

final_score: rating: 7: Good paper, accept
review_content: This paper leverages model predictive control (MPI) and proposes an approach which utilizes a transition+reward model, expected return model, and behavior-cloned policy to plan in a fully offline setting. The authors conduct experiments and ablation studies, showing the benefit of their approach in small data regimes and transferability to secondary objectives (e.g., constraints).

Positive feedback:

1. MBOP is an interesting approach to utilizing environmental model together with a behaviour-cloned policy in an MPI framework. Its main novelties w.r.t PDDM include the use of the behavior-cloned policy and an expected return model. 
2. MBOP shows to be a strong baseline for offline reinforcement learning, especially in small data regimes, which is especially interesting. Their ablation studies show the importance of using behavior cloning.
3. Unlike other methods for offline-RL which first learn models and then use them in to solve the underlying MDP, MDOP in its essence allows to learn secondary objectives without the need to retrain the policy on the new MDP. The constrained objectives are especially interesting with this respect.

Questions and concerns:

1. It was unclear (even after reading the information in the appendix) how the models f_m, f_R and f_b are trained (e.g., when and if ensembles are used, etc.)

2. The method is fundamentally different than MOPO and MBPO, and it not clear if the reason for improvements are due to fundamental algorithmic factors. Some examples:
- The authors use the same model head for simulating the full trajectory. 
- The use the mean reward and not the maximum reward as in MOPO. MBOP also learns an expected return model in addition to the reward model in MOPO. 
- Are the same transition models used in MBOP as in MOPO and MBPO? 
Such factors may greatly affect performance and overall results. If these are the factors that affected the boost in performance, it would greatly reduce the general interest of the proposed approach. It is thus currently unclear if the comparison is fair.

3. The authors chose to discard episodes that are below a certain threshold for the training of f_b and f_R. This raises a question as to the ability of the algorithm to utilize non-optimal data, e.g., dirty data that wasn’t collected by a quasi-optimal policy, or a mixture of policies. 

4. As MBOP utilizes behavioral cloning at its core, it would be interesting to see how it compares to behaviour regularized offline reinforcement learning: https://arxiv.org/pdf/1911.11361.pdf

5. While I understand this work is a generalization of previously developed methods (MPPI, PDDM), some of its underlying assumptions are unclear. In MBOP actions are linearly interpolated (beta mixtures and weighted trajectories). When and why do the authors assume such interpolation should work? 

6. There are two flaws that make the applicability of MBOP somewhat problematic:

- Lots of hyper parameters - unclear how these should be chosen. This includes the many different models which are not fully discussed.

- Uncertainty of the algorithm is not taken into account by MBOP. I believe this to be an essential requirement for an offline-RL algorithm to be applicable, as no new experience can be collected.

7. Finally, the clarity of the paper could greatly be improved. Overall it feels like the paper is not self contained and hard to understand. Many notations are not explained, and some of the taken approaches are unclear. I feel that Algorithm 2 should be explained better. The authors should provide more detailed information on what each line signifies and why it is an important feature of their approach. 
Minor question: What does "min" signify in T_{min} in the Algorithm 2, line 11?



Typos:

Section 2.4:

- “…is a truncated value function, which *provides* the expected”

Section 4.3: 

- “*One* of the main…”
- “We can *then* adapt the trajectory…”

review_title: Interesting approach for planning in offline RL - allowing for small data regimes and good transfer

Meta-review: # Quality:
The experimental evaluation is thorough and well designed. 

# Clarity:
After rebuttal, the paper is well written and clear in its contributions.

# Originality:
The proposed approach builds over existing literature, as clearly indicated in the manuscript and acknowledged by the authors in the rebuttal, by mixing existing contributions in a novel fashion.

# Significance of this work: 
This work deal with an important and timely topic. The experimental results are convincing and demonstrate strong performance for the proposed approach.

# Overall:
This manuscript provides an incremental but solid contribution to the topic of model-based reinforcement learning.

# Personal comments:
- I disagree with Reviewer2, in that I feel that after the rebuttal the manuscript clearly and thoroughly states prior works and the novelty of the proposed approach. This is not a survey paper and should be hold to realistic standards.
Decision: Accept (Poster)