Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper addresses the problem of sample complexity in reinforcement learning. The approach is to drive down the sample complexity using reward shaping, where the reward shaping function is given by a planner. The paper shows how to use final-volume-preserving
reward shaping (FV-RS) to convert a plan into a reward shaping function. The approach is evaluated on a simple pushing domain against planning in two forms, and Hindsight Experience Replay. The L2E approach proposed here outperforms the comparison algorithms. 

main_review: Overall, I like the idea of this paper and would argue for its acceptance. The technical ideas are strong, and the paper is (mostly) well-written. The primary limitation of this work is the experimental evaluation, especially in an increasingly crowded field of prior work combining planning and learning.  It is not clear to me that the most relevant baseline is HER. There are a number of more relevant works, such as PRM-RL (Faust et al, 2014) and Hoel et al (2020), both of which explicitly combine planning and learning for exactly the same purpose as L2E, and there are other references as well. I think the real novelty is the use of FV-RS as the shaping function, but the other papers in this line of research need to be considered. 

I would recommend substantially strengthening the experimental evaluation. The single domain of box pushing is not particularly compelling, and does not show off the strength of incorporating a planner in the reward shaping function — what would be better would be problems with a much longer horizon and places where a significant deviation from a greedy strategy would be important, but with stochasticity to be avoided. (BTW: the plan shown in figure 1 was very hard to interpret. I now realize that the reason the end effector is changing height is because it is going over the box to change sides to push, but it took quite a while to figure that out. Some more explanatory text would help considerably. Maybe also draw the box at different along the trajectory with a significant alpha channel value.) Figure 1b was not a particularly effective way of communicating the results. It was not clear to me if these are the average and variances of the 10 agents on one problem, or the average and variances of the 10 agents times 30 random start/goal states per agent. 

Even for these experiments, I would recommend running L2E and HER longer — it is not clear to me that L2E $ S^{bias} _{10,1000}$ has converged, especially when looking at 1d. I assume the loss in performance for 10 plans from 1000 samples is because it has experienced fewer transitions. And if indeed L2E $ S^{bias} _{10,1000}$ has converged, I would want to know why it is not exploring further. 

Equation 3 is very important, and it is confusing and feels somewhat arbitrary — the paper relies too much on explication from the FV-RS paper, including replicating notation without explanation. The fact that f(p) is the final state of the plan is never stated, although the fact that f(p) is the goal on the following page allows the reader work backwards. $p_{k(s)}$ should be explained, as well as the fact that the subscript is indexing the plan state. Even with the the FV-RS paper in hand, the justification for the $k(s)+1/L$ term is unclear to me. I would encourage the authors to add explanatory text for the terms in the FV-RS reward shaping equation. 
limitations_and_societal_impact: I appreciated that the author checklist pointed at the specific section where they were attempting to discuss limitations. Of the papers that I reviewed, this paper probably provided the best analysis of the limitations of the work. That being said, the discussion of limitations was quite cursory. There is room for considerably more analysis of the limitations of this work, especially in the experimental stage. It seems clear that the biggest limitation of the work is the loss in optimality induced by FV-RS. I expect this loss in optimality will create very negative consequences when the optimal plan and optimal policy are wildly divergent. 

For instance, imagine a world with two routes to the goal, where one route is highly stochastic with very high expected cost, but is much shorter, e.g., a short cut along a cliff’s edge. The other route is much longer is far less stochastic (or just has no negative outcomes), so much lower expected cost (a route away from the cliff’s edge). if the planner assumes a deterministic world, it may incorrectly assume that the cost of the path along the cliff’s edge is much lower than the cost of the longer route, and very strongly bias the learner in the wrong direction. 

This is not to say that this paper is wrong or incorrect in some way, but that obvious limitation does have to be addressed. 

The author checklist indicated that the authors did not discuss any potential negative societal impacts of the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes Learning to Execute (L2E), which can make use of a pre-designed planner in reinforcement learning (RL) in robotics. The authors extend the idea of final-volume-preserving reward shaping (FV-RS) and formulate plan-conditioned MDPs for designing L2E. The method was evaluated through a simulation experiment.
main_review: The paper has certain originality regarding the idea of introducing FV-RS, especially from the theoretical viewpoint.
However, the validity of this method was not demonstrated sufficiently. Comparison with more pre-existing methods is expected. 

The quality of this paper is generally reasonable.
The clarity of this paper is also reasonable. 

However, the contribution is not significant. It is incremental because it is a straightforward application of FV-RS. For that application, some assumptions, which seem unrealistic in the real-world robotics environment, are introduced.  This point relates to the definition of "model-based planner" in this paper, which will be discussed in this review's "limitation" part. 
At least, the significance is not demonstrated sufficiently through the experiment.
limitations_and_societal_impact: The paper is motivated by the real-world robot application of RL.  (At least, the authors insisted so.) Unfortunately, the approach and discussion do not seem to grasp the nature of the real world and actual situations in robotics. That seems to be noted to start the discussion of the sample efficiency. A critical issue of this paper is the unclear definition of "model-based planner." It is quite unclear to me. What kind of controller system and settings in robotics do the authors assume?  As far as I read through the paper, it is some idealistic Mathematical element. 
Specifically, I am not sure what kind of "model-based planner" can satisfy the condition (5). The goal of a planner (or a controller) in state space is not always available in robotics. At least, the uncertainty of the environment and some clear and realistic assumptions should be considered.   Of course, in a simulation environment and a simple scenario like an experiment in this paper (which we may call a "toy problem"), we can find a setting satisfying the condition. If "model-based planner" in robotics does not satisfy the condition (5), the approach is not valid for solved the problem raised at the beginning of this paper.  
Also, the term "model-based" is also used in "model-based RL," and policy is also called planner in some context. From these views, the authors should clarify the definition of "model-based planner" in this paper and show examples of them in robotics.

The comparison is limited. The method is about the integration of a pre-designed policy and RL via sub-reward. In comparison, I recommend the authors to compare the proposed method using other types of sub-reward approach, e.g., [Andrew 1999].

Also, this study is closely related to the integration of imitation learning (IL) (i.e., learning from demonstration or expert information) and reinforcement learning. For example, the following paper integrates IL and RL via sub-reward based on control as an inference framework. The relationship is worth mentioning. 

- Akira Kinose, Tadahiro Taniguchi, Integration of imitation learning using GAIL and reinforcement learning using task-achievement rewards via probabilistic graphical model, Advanced Robotics, 34(16), pp.1055-1067, 2020. DOI: 10.1080/01691864.2020.1778521
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method for combining plan-conditioned policies with reward-shaping and a given approximate planner to a sparse-reward, seemingly challenging robotic manipulation task. However, the paper is lacking in experiments, which is the main reason I am voting for rejection.
main_review: ## Paper Strengths
**Intuitive Method**
This paper demonstrates that a plan-conditioned policy combined with an FV-RS reward shaping function is able to allow for better performance on goal-conditioned, sparse reward tasks. The method is novel, although the contribution isn’t too surprising.

**Hyperparameter Tuning**
The authors perform a proper hyperparameter tuning scheme for HER when comparing against the baseline.

**Performance Improvement**
The method presents a modest performance improvement over the baselines.

## Paper Weaknesses
**One environment, one task**
The authors should evaluate on at least 2, preferably 3+ environments/tasks to truly demonstrate the advantage of their method. This is one large reason I am currently not voting for acceptance.

**Method Clarity**
Question: Are plans embedded or given in full to the plan-conditioned policy?

**Ablation Studies**
How does planner quality affect L2E? How do plan lengths affect L2E? Can you show ablations here?

**FV-RS Introduction**
The $F_{FV}$ reward is discussed but not given an intuitive explanation. This section, and the method in general, would be more clear if all 3 terms were explained intuitively in Eq.3 and their effects on policy behavior were discussed explicitly.

**Missing Related Works**
Combining model-free and model-based RL is presented as a contribution, however there are works that have done this before (see for example, “When to Trust Your Model: Model-Based Policy Optimization by Janner et al, and “Temporal Difference Models” by Pong et al). The authors should add citations to these works and contrast L2E with them.  

limitations_and_societal_impact: L2E limitations are actually not discussed, despite a reference to the discussion in the checklist.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes to condition policies on plans. Therefore, the policy learns to execute an approximate plan and correct it if necessary. This idea is instantiated in an algorithm that uses SAC for policy optimization and a "crude manhattan-like planner" for planning. The algorithm is evaluated on a box pushing task, where an improvement in sample-efficiency over goal-condition RL is shown, and an improvement in success rate compared to model-based plan execution with learned inverse dynamics model is demonstrated.
main_review: The idea of conditioning policies on expert plans as presented in this paper is novel according to my knowledge. The paper is of good quality and it is written clearly. Together with the code in the supplementary materials, the paper provides sufficient details to reproduce the results. However, the evaluation is performed only in one environment and there is no discussion or evaluation of the influence of the function that encodes the plans before passing them to the policy. This lack of evaluations makes it hard to judge the significance of the proposed method as it is not clear if it generalizes to other settings. Since the main competitive approach considered in the paper is HER, it might make sense to add evaluations of the proposed L2E algorithm in the environments from the original HER paper.

Major comments
- line 231: "we analytically encode the plans into a 4D latent space". Provide the details of this mapping in the paper since this is one of the crucial elements of the proposed approach. Why 4D? What properties should this mapping satisfy? Does the dimensionality of the latent space depend on the task? Either theoretical study or empirical evaluations of the choices of this mapping should definitely be provided, because conditioning the policy on plans instead of goals is the main difference of the proposed method  compared to goal-conditioned RL.
- lines  272-276 say that L2E agent significantly deviates from the plans. This appears to imply that conditioning on the plan may not be so informative to the agent. What is the advantage of having a plan-conditioned policy instead of a goal-conditioned one? It is shown in the pushing experiment that with the chosen hyperparameters one achieves faster convergence, but are there any benefits in general that one should expect from feeding in plans instead of goals?
- it would be beneficial to the reader to contrast the proposed approach to guided policy search (GPS) and DeepMimic, as those methods also involve imitation of an imperfect teacher, although they don't explicitly feed the plans as input to the policy

Minor comments
- line 178: plan replay strategy S_n is defined as "a distribution over n replay plans". However, in lines 181-182 it is said that the uniform replay strategy samples n unique plans uniformly from the replay buffer D, which contains more than n plans. Therefore, I think the authors mean that S_n is a distribution over integers 1, 2, ..., len(D). So, the subscript n seems misleading in S_n because the distribution itself does not depend on how many samples are drawn from it.
- lines 198-199: what is meant by "unstable dynamics" in the pushing task? Every state of the system seems to be a stable equilibrium state if no control input is applied. Maybe a different term instead of "unstable" would be more appropriate here. The same in line 285.
- typo:  line 146, policy should be conditioned on state instead of action

Comments after rebuttal
I thank the authors for addressing my questions and  for providing additional experiments. I raise my score from 5 to 6. There are still quite a few concerns raised by other reviewers, such as providing comparisons to more directly related learning+planning algorithms instead of HER. Furthermore, the environment used for additional experiments is still a custom made environment, therefore it is hard to directly relate it to other papers. I am also not convinced by the argument that having a plan-conditioned policy is necessary to have a multi-modal policy: in soft Q-learning this is achieved by using an energy-based model for the policy, and if it is goal-conditioned, it may produce different solutions as samples from a multimodal distribution. For all these reasons, I can only say that the paper is marginally above the acceptance threshold.

Final comments
The authors addressed all my concerns and promised to add the results provided in the anonymous github repository to the paper. With these additions, I consider the paper appropriate for publication and raise my score to 7.
limitations_and_societal_impact: -
review_title: null

Meta-review: The paper combines learning and planning to increase data efficiency, motivated by robotics application scenarios. In more detail, the authors leverage the fact that in such scenarios, often, a coarse model is available, that allows for making a coarse, but probably suboptimal plan. The authors propose using this plan to provide shaping rewards to a RL agent that is then free to further optimize the policy. For this, they build on "plan based final volume preserving reward shaping" introduced by Schubert et al. The main contribution of the authors is to
- Allow policies that generalize over instances by conditioning on the plan
- Using this formulation to introduce plan-replay-strategies. 

Initially, the reviews was mixed, but after the discussion phase all reviewers recommend acceptance. The main reasons being: 
- Strong and original technical ideas
- Mostly well-written
- The experimental evaluation was initially somewhat limited, but during the discussion phase the authors have made these stronger by adding another environment and clarifying the baseline methods used. 
- The methods limitations could be more explicitly discussed. 

I'd like to add that I think that the authors could more explicitly discuss what is novel in their paper compared to the earlier Schubert paper. E.g., the FV-RS is directly introduced in a 'universal' formulation (3), making the transition to universal policies and reward shaping functions more smooth but also making this part of their contribution less explicit. 
Decision: Accept (Poster)