Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors  highlight the problem of iterated batch RL (and thus its deployment efficency) and propose an algorithm along
with novel evaluation schemes.

There are several things I like about this paper:

- The topic is very important and underexplored, especially the deployment efficency w.r.t. the number of batch collections
- Lots of relevant work is cited
- The experiment make sense given the research question. Especially evaluations such as in Figure 2.
- The results show significant improvements of the proposed method over state of the art
- The method  is "simple", in a good way, and has not that much moving pieces, compared to other RL algorithms

The are a few things I think could be improved:

- Because the paper considers "repeated batch" it should touch the subject of "if I can collect x batches, what batches would tell me the most?". I.e the algorithm as written right now is "greedy" in the sense that the policy will only act in the way it considers most optimal (and is not completely different from the behavior policy).  Batch exploration strategies would be highly interesting
and as "iterative batch RL" is the domain of this paper, these need to be addressed.

- Related:  While epistemic uncertainty is modeled (via Ensembles of dynamic models) it is not explicitly utilized.  Eq. (4) contains a "safety mechanism" via the trust region approach.  Why not include a risk criterion over the ensemble spread?

- Why use only deterministic models?  Using something like [1,2] that enables modeling stochastic effects would improve the 
universality of the approach further, not being restricted to deterministic problems.

I am a bit unsure about the novelty of Section 4  --  most pieces of the algorithm are already known or "obvious".  To me the 
biggest novelty lies in identifying the "repeated batch" as  an important problem, that is underexplored and shows how 
classical RL method underperform strongly here. From this  evaluations like shown in Figure 2 are novel and make a lot of sense. 


[1] Chua, Kurtland, et al. "Deep reinforcement learning in a handful of trials using probabilistic dynamics models." Advances in Neural Information Processing

[2] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. "Simple and scalable predictive uncertainty estimation using deep ensembles." Advances in neural information processing systems. 2017.

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper has been well written and is easy to understand. The premise of the paper is clear, and methods have been evaluated properly. 

Suggestions to improve the paper:
- While the deployment efficiency metric has been clearly explained, the justification of this metric has not been elaborated on sufficiently. For the community to adopt a new metric, the paper needs to provide enough evidence to show that deploying policy is expensive, time consuming or requires manual curation. What are the steps involved in deployment? In what situations is deployment expensive? How common are they? Sample efficiency is a well established metric because it makes sense that we need to learn a good policy in as few real world interactions as possible. One can argue that safe exploration is essential for real world deployment. However, that is neither discussed nor compared with. 
- The lack of clarity in the deployment efficiency metric also appears in the evaluation. Why does a batch size of 100K make sense? Why limit it to 5 to 10 deployments? These numbers appear to be arbitrary. 
- How does one decide the trade off between sample efficiency and deployment efficiency? 
-  A clarification question: How is the 50k and 100k transition obtained for evaluating sample efficiency? Is it sampled randomly from the 1M dataset or is it sliced without shuffling?
- The BREMEN algorithm borrows ideas from related prior works, and is reasonable. However, it is unclear why the algorithm is better suited to improve deployment efficiency compared to prior works. From the explanation provided in the paper, BREMEN is a good offline RL algorithm and is coincidentally better at deployment efficiency as well. 


review_title: Justification of Deployment Efficiency metric is unclear

final_score: rating: 7: Good paper, accept
review_content: Summary

This paper proposes a new approach to learning control policies with improved data efficiency and fewer number of data collection sessions (with each session using a different policy). Further, the authors proposed a new concept of “deployment efficiency”, with a new “deployment” referring to using a new policy to interact with the real environment, for example, for data collection. The new approach belongs to the family of model-based online reinforcement learning algorithms and seems to primarily augment a prior approach, called ME-TRPO, by using a helper policy trained by behavior cloning data collected after the most recent deployment of learner policy. The experiment results validate that the proposed approach achieves better data efficiency and deployment efficiency compared to prior approaches.

==============

Positives

The experiments are very extensive and of high quality. A diverse set of tasks is used, along with a good range of prior approaches for comparison. The experiments are also well designed,   with one focusing on data efficiency in offline setting and the other one mimicking practical usage in online setting with limited number of deployments.

The experiment results are clear and convincing. Performance is the best in almost all tasks. 

It is a great step towards wider application of RL algorithms in the real-world by recognizing practically important metric such as deployment efficiency.

The authors did a good job in introducing the varied set of related works and the general background, providing a clear organization and classification of prior approaches.

==============

Negatives

The authors could make a clear statement on the contributions of the paper. Several features of the proposed algorithm can be found in the prior approach of ME-TRPO, including all steps shown in Algorithm 1 except steps 2 & 5 & 6, i.e.: the use of an ensemble of learnt dynamics models to mitigate model bias and training instability, the use of imaginary rollout, the choice of TRPO for policy optimization, all are present in ME-TRPO. The clear innovation in this paper seems to be in using the behavior policy to bootstrap policy learning in a new iteration of deployment. The authors could make clear the approach is an extension of ME-TRPO, which at present is not in the way the authors proposed the algorithm.

While I don’t dispute the merit of deployment efficiency, it can be better motivated. The first paragraph of Section 3 mentions “verify each policy before deployment”; it’d be good to show references of real world applications of this procedure. It’s conceivable in practical settings, there can be better ways to achieve the verification in a way that circumvents the cost of deployment (for example, limiting policy outputs to a safe range at runtime, instead of “checking out-of-bounds actions” as a way of pre-validation).

==============

Recommendation

Overall the proposed approach clearly yields empirical advantages of prior approaches in both data efficiency and the newly proposed metric of deployment efficiency. The paper suffers from lack of algorithmic novelty, but the experimental efficiency outweighs. 

review_title: Official Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper explores an under-researched problem, that of minimizing the number of policy updates in an RL setting (or being “deployment efficient”). This is an important aspect of using RL agents in real “production” environments where there may be many reasons why updates are costly and limiting them is an important consideration in the choice of RL method (or whether to even use RL).

The paper shows that so-called "off-policy” methods which, by their naming as such, it is implied that they should work in a sparse-deployments environment are, in fact, not suited (and often not evaluated) for this regime.

By introducing a set of simple and strait-forward steps to the update and deployment process, the paper shows performance that approaches the continuous-deployment performance of comparable un-constrained methods.

The main ingredients of the proposed method seem to be:
Model based approach to support model-based offline exploration (an ensemble of models to prevent exploitation of model inaccuracies)
Re-estimation of a model by Behaviour Cloning (BC) with data from last deployment (appears to have a large contribution though I don’t understand why)
Conservative off-line policy updates (constrained by KL to BC policy) using offline rollouts (with forward model ensemble)

The evaluation presented in the paper and extensive (13-page) appendix clearly show the advantage of the proposed method in the sparse-deployments regime and the overall competitive performance with regards to sample efficiency as well. Code was made available as supplementary material.

I am unclear on why each training iteration should start with a policy learned from behavioural cloning of the last deployment’s data instead of the model that was deployed which would be available at that time. Figure 4 clearly shows BC to be the better approach In practice but I would appreciate some intuitive reasoning for this (unless this is standard practice).

Perhaps my main concern with this paper, given the problem it addresses - that of a real deployment setting, is that it seems that some of the parameters that need to be tuned to achieve great performance require fitting to the specific task (i.e. deployments !). As far as I understand it, parameters such as the number of offline policy updates per deployment or the weight of the KL-divergence in the policy update step are crucial to good performance yet the paper does not explain how to choose them without engaging in the true environment. If that is correct than this seems to defeat the aims of the paper and question the overall methodology for deployment-efficiency.

review_title: simple yet effective method for achieving deployment-efficiency (but requiring some prior knowledge?)

Meta-review: # Quality:
The algorithm is thoroughly evaluated and several interesting experiments are included in the appendix. 

# Clarity:
The paper is generally well written.

# Originality:
The proposed approach is a small but novel improvement over existing algorithms (to the best of the reviewers and my knowledge). The concept of "deployment-efficiency" is, in my opinion not novel, since it seems mostly a rebranding of what the MBRL community traditionally refers to as "data-efficiency" -- although I agree that deployment-efficiency is indeed a more accurate term.

# Significance of this work: 
The paper deal with a relevant and timely topic. However, the paper does not compare to the larger MBRL literature. Hence, it is difficult to gauge the significance of this work.

# Overall:
This manuscript offers a good contribution to the topic of model-based reinforcement learning algorithms.

# Minor comments: 
- I suggest removing the word "impressive" from the abstract. This is a subjective term, which should be avoided.
- In my personal opinion, it would be nice to include experiments with more state-of-the-art baselines such as PETS and POPLIN, for which code is available online. It is unclear to me how much the improvement in performance depends on the algorithm itself compared to just having larger batch sizes. From this perspective, Figure 5 in Appendix B is probably the most interesting insight of the manuscript, to me.
Decision: Accept (Poster)