Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper's goal is to evaluate the sample-efficiency of planning-based and policy-based approaches, as well as hybrid versions of each, in several multi-task continuous control environments. The paper shows that for the tasks investigated and the specific computational budget allocated to the planner, the planner performs significantly worse without a good proposal distribution (policy sampler), even when employed with ground-truth access to dynamics and rewards. Then, the paper presents multi-task experiments with variations of their proposed hybrid planning-amortized method that concurrently learns a proposal distribution, model, and critic; these experiments show that the MPC-based variants outperformed a model-free method to varying degrees.
main_review: Strengths
--
- The paper investigates a topic that is interesting and timely
- The results illustrate settings in which the quality of the policy proposals are very important to the final performance
- The proposed hybrid method is straightforward
- The paper includes exhaustive ablations

Weaknesses
--
- W1 Some writing issues: **W1.1 Missing proposal terminology** and **W1.2 Dubious justification of model determinism**
- W2 The presented evidence doesn't fully validate the claims: **W2.1 Missing evidence of computational burden**,  **W2.2 Narrow set of environments used**, and **W2.3 Narrow set of tasks used**

Weakness 1:
--
-  **W1.1 Missing proposal terminology** The terminology "proposal" used in the abstract and introduction needs a clear definition if it's going to be used here. Is it a single action sampled from a policy? Is it a stochastic policy? Unfortunately, the current introduction forces the reader to guess.
- **W1.2 Dubious justification of model determinism** S4.2 The use of deterministic models here is fine, but I'd argue that it's a reasonable assumption mainly because of determinism (I assume) in the true environment dynamics. If the MDPs investigated had significant stochasticity in the environment dynamics, then perhaps it would not be a great assumption, and performance would be significantly worse. S4.2 should be explicit about the degree of stochasticity in the actual environment dynamics. If the degree is insignificant (or none), it'd be great to get the authors' take on whether they agree that the main reason deterministic dynamics models work well is simply that the modeling class can represent the true dynamics models employed in the target MDPs.

Weakness 2:
--
- **W2.1 Missing evidence of computational burden** S4.3 The claim that planning in these tasks cannot be performed alone with the ground-truth dynamics model and reward function requires more detail and requires more evidence. It is fair to assume that with enough search time/compute, a long-enough planning horizon, and access to ground-truth dynamics and rewards, the tasks can be solved e.g. via brute force search. Thus, the claim needs to be more specific about the computational restrictions that necessitate the use of a proposal-generating policy. This issue also appears in the first paragraph of S6 -- there's no actual discussion or experimentation of what constitutes 'limited' computation time. Because this claim is central to the paper's message -- that a good dynamics model is insufficient for transfer -- the paper would be much stronger if it presented evidence across a clear regime of computational time constraints that the result holds. For example, a plot of Performance vs. Computation time, with the 'optimal' and 'best proposal-based method' performance as single points (or sets of points), accompanied by the performance of MPC with GT dynamics and rewards with different computational budgets plotted as a curve. By the paper's claims, the latter MPC curve should be significantly below the 'optimal' and 'best proposal-based method' for reasonable computational regimes, and above the 'best proposal-based method' when an 'unreasonable' amount of compute is assumed (if feasible).
- **W2.2 Narrow set of environments used** The employed set of multi-task evaluation environments in the paper is quite small. While it's good that the single-task environments are presented in the appendix, without a broader set of multi-task environments, we cannot draw broad conclusions about these methods in other multi-task settings
- **W2.3 Narrow set of tasks used** The multi-task evaluation is limited to tasks in which the multi-task nature is target-goal reaching. Results presented on different MDPs in which the reward function variation is more sophisticated than goal reaching would provide more support for the claims.

Minor weaknesses
--
- The DAGGER reference in 4.1 is too tangential (neither necessary nor particularly useful).
- S4.3 Near "We can then improve our proposal by minimizing the KL divergence", the optimization variables need to be clarified, since it's unclear if the $\pi_\theta$ contained within $\tilde{\pi}_{\mathcal B}$ is optimized. However, Eq. 1 clarifies this, so perhaps just call it a 'forward' KL divergence.
- S4.1 says more details about the planner are in Sec B of the supplement, but the supplement didn't contain a .pdf (I found Sec B in the appendix of the .pdf)
- Fig 2 needs more details in the caption or in the main text to describe the difference between 'actor' and 'proposal'. My understanding is that the 'actor' is the MPC actor if MPC is used, or equivalent to the policy (and the 'proposal') if MPC is not used.
- Footnote 2 requires a citation or evidence for the claim "MPO has a known problem with shrinking policy variances". Furthermore, this hypothesis could be tested simply by adding a policy-entropy bonus to objective function, no?

summary_of_the_review: While the research question and initial evidence are promising, the main issue with the paper is Weakness 2 above, particularly W2.1. Evidence is required to support the claimed computational burden of planning with ground-truth dynamics model and rewards. Furthermore, the multi-task experimental results are only conducted in two environments (W2.2) and only for goal-directed tasks (W2.3), which doesn't support the claim that the quality of the proposal distribution is critical to planning in *general* **multi-task** settings; it instead only constitutes evidence that the quality of the proposal distribution is critical to planning in *some* **goal-directed tasks**. 

In my opinion, this missing evidence weakens the paper slightly below the bar of acceptability.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper performs a careful study of the advantages of integrating model-based components into an RL system.  In particular, they clearly distinguish between the effects of learning a predictive model and learning a proposal distribution for a forward-sampling planner.   They sensibly focus on what seems like the "sweet spot" for model-based methods:  transfer of a model from one task to another in a domain with the same basic dynamics.   They find that, when using forward-sampling planners, the proposal distribution is the driving factor of success, so that learning and transferring a good model does not, in itself, improve performance much.
main_review: This paper is very clearly written, well argued, and well executed.    The experiments are well-designed and clearly illuminate the points being made.  The paper adds an increment to the collective knowledge of the field about RL methods.

I also find it deeply unexciting and not at all imaginative.    I would have been more enthusiastic about a less well executed paper with a really new idea.   Surely the fact that transfer isn't working very well tells us that we need some new insights (or I guess, an argument about why all of our intuition about transfer in problems like these is wrong).   Is there a way to improve the planner so that it's less stupid, for example by learning landmarks or making it hierarchical?  Could we make the proposal distribution depend on the goal in the GTTP tasks?

I have some small comments/questions, but none of them are critical:
- In the early parts of the paper it would be good to clarify what you mean by "learning proposals" (it's made clear later)
- I found it jarring that the word "proposal" is used frequently when really "proposal distribution" is what is intended;  to me a "proposal" would be a single sample from a "proposal distribution."
- When you discuss "target poses" do you really mean "target configurations"?   (This is ambiguous, I guess, but in robotics it's most common to use "pose" for a 6DOF pose of a rigid object and "configuration" for anything more complicated.   
- Do you think anything different would have happened if the goals were a bit more naturalistic (e.g., get the centroid of the agent into a region)---it's rare in any natural situation or application that the entire configuration of a complex agent would need to be copied.
- The design of the model was interesting---I was eager to read more about it in the paper body.
- I am concerned that stochasticity was "waved away" without enough consideration.  In many important domains, the distribution of possible outcomes is not at all well summarized by the mean.  It seems important to at least acknowledge this.
- It was a bit confusing to read the phrase "task-agnostic proposal pre-trained with a behavioral-cloning objective."   It's hard to imagine a situation in which behavioral-cloning is task-agnostic unless, for example, you train on BC data from a whole mixture of tasks.  Perhaps that's what you did.

(To the degree that I lack confidence it is with respect to the detailed novelty of this paper---I don't keep up with this sub-part of the literature and so I can't attest to whether or not this was already all well-known, for example.  But I think I understand the methods described at a sufficient level and get the experiments and their overall message.)
summary_of_the_review: Whether or not to accept this paper depends on what we think conference publications are for.  

This paper contributes some clear and well justified knowledge.  But the increment is small and it seems likely to me that substantially new and different techniques will arise soon and render this irrelevant.    It seems unlikely that this paper will inspire or excite new research.

Ultimately, this decision is above my pay grade, and depends on the program chairs' views about the role of paper acceptance.

My guess is that a good but not hugely exciting paper is thought to be on the positive side, so I'll lean positive.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides a thorough, detailed evaluation of model-based RL, as realized via MPC-based planning.
In particular, it examines: (a) how it can be improved via learned policies that act as MPC proposal distributions;
and (b) to understand and disentangle the benefits of (i) model transfer and (ii) proposal transfer,
when using model-based RL. Comparisons are performed using both CEM and SMC samplers. 
For the locomotion-related benchmarks considered, MPC with a learned model & proposal can provide modest improvements
wrt a well-tuned model-free baseline (MPO). However, for a more challenging go-to-target-pose (GTTP) task,
they offer significantly improved learning speed and final performance.
The benefits of model-transfer are rather marginal, even when transferred to the same task, 
counter-to-common-belief-and-intuition. 

main_review: Strengths:
- demonstrates the benefit of combined model-learning and proposal learning, in support of MPC, on simple & a challenging GTTP task
- provides a greater understanding about model-transfer and planner-transfer, across tasks
- makes us rethink some of the conventional assumptions about model-based RL

Weaknesses:
- only MPC-settings are considered;  in an ideal world, the reader would also be able
  to understand how these methods stack up against model-based methods that use the model
  in different ways.
- the paper could comment on the possible relative benefits retained by MPC approaches in general, 
  over policy-based methods, in terms of adapting to out-of-distribution states.
  

The paper can in part be seen as an "understanding" paper, one that provides insights into
the interplay between planning and model-free control, and how they can be integrated.
This remains and underexplored area.  Relatedly, the relative benefits of model-learning
in task-specific and for new-tasks with shared dynamics also needs to be better understood, 
as noted by the paper.

The paper builds on an interesting parameterized task, the go-to-target-pose (GTTP).
Such parameterized-goal tasks are likely to provide more insight than fixed tasks.

I'd love to see a structured abstract diagram or table that could somehow capture the design space
of model-based planning methods, and the general assumptions that motivate the different design choices.
Caveat:  I don't know if such a diagram is realizable.

re: section 2, other uses of models
Perhaps older refs to backprop-through-time (BPTT) would help communicate that BPTT has been
used even before the rediscovery of backprop, i.e., state adjoints used in control.  Or even just go back
to the Nguyen & Widrow "Truck Backer-upper" from 1990.

"it is possible to train policies on model rollouts to improve data efficiency (Janner et al)"
It might make more sense to simply cite DYNA here, as one among many?

The connections between MPO and SAC could be made more explicit.
Or the differences, if these are relevant for the results presented in the paper.

The idea of separating out proprioceptive information from other goal-related state information
is a good one.  It is in some sense specific to a certain class of MDP, however, so that could
be worthwhile clarifying (although will be obvious to those working on robotics. It also points
to some form of control hierarchy, where new tasks won't need to leverage significantly new proprioceptive states
and actions.  Is the global orientation wrt vertical i.e., basic IMU info, also modeled as being part of the proprioceptive information?
(ok, I now see the answer is "yes" in App A).

Figure 2:  The middle column of graphs, giving Target pose counts, would be better replaced
by simply another task.  The results are highly correlated with the reward graphs in the first column,
and a tired reader will mistake the results as being yet more training curve graphs. 

Figure 3:  The overall title of "Model and/or Proposal Transfer -- Actor Performance" is confusing.
And perhaps the legend could be improved, i.e., MO+BC could be relabeled as "MO+BC (no model)"
and "Proposal from scratch" could be relabeled "model transfer"; and "Reloaded Proposal" could become "Proposal Transfer".

It could be Worthwhile noting in the main text that SMC is more conputationally efficient than CEM.
This detail is currently in App E.1, but would be worthwhile being part of the Table 1 discussion.
summary_of_the_review: This is a reasonably thorough evaluation of common assumptions related to model-based RL, particularly in the context of MPC methods. The ideas are tested on a range of problems, ranging from the simple to the challenging, i.e., the go-to-target-pose (GTTP) task. The documented benefits of learned proposals on difficult tasks such as GTTP, as evaluated using both CEM and SMC, are also a worthwhile contribution.  The empirical work is solid.  A better understanding of model-based methods in continuous control settings is important for the RL community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
In this paper, the authors follow the line of research of adding policy in online model-planning.
They consider multi-task / multi-goal environments, and use MPO as the proposal network, which improves the performance.

main_review: 
Pros:
1. The quality of the experiment section is very good.
There are proper baselines, and the environments are well designed.
The videos are also very good, providing ideas how well the algorithms are.

The difficulty of the environments are much higher than the ones we currently use for model-based algorithms.
I think they can be good baselines if the code is released along with the algorithm as a default baseline for future research.

2. The related work section is very adequate.

Cons:

1. Some details of the paper are missing.
And since MPO consists of a very important role in the paper, it would be necessary to include a preliminary section.

2. No code is released, and it is almost impossible to reproduce the results presented in the paper.
For the reproductivity, I don't think other researchers can reproduce the algorithms without access to the engineering details.

3. It seems that the most obvious novelty of the paper is the use of MPO as the proposal policy network.
It seems rather arbitrary and I wonder how other model-free algorithms would have fit in the framework.

summary_of_the_review: 
I think in general the algorithm is very good. The environments are pretty interesting.
The release of code will be quite important to this project if it aims to encourage future research.
review_title: null

Meta-review: The paper examines the advantage of using models in RL.  The authors' rebuttals convinced us of the value of the paper.
Decision: Accept (Poster)