Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper aims to solve long horizon state-based robotic tasks by leveraging RRT-style tree search with goal conditioned policy and task-dynamics model. The goal conditioned policy is learned from play-data using variational learning. It is able to solve subtasks with a shorter task horizon within a longer-horizon task. Simultaneously, the paper learns a task dynamics model which takes in latent task parameters to predict the final state after completion of the task. Finally, the paper uses a task dynamics model with RRT-style tree search to find the sequence of subtasks needed to complete the task and then solves each of the subtasks with the goal conditioned policy.
strengths_and_weaknesses: Strengths: 
The proposed technique is able to nicely solve hard long horizon tasks. For context, it is very hard to solve these tasks with existing deep RL (model free or model based) techniques with end to end training. The paper does a good job of leveraging past data to learn a goal conditioned policy as well as a task dynamics model and then using them to plan for longer horizon tasks. 

Weaknesses:
The writing could have been a bit more clear with respect to the metrics reported. I don’t completely understand the difference between solution found and feasibility. By solution found, do we mean the agent is able to find some solution using the dynamics model (but that solution could be totally wrong if the dynamics model is bad)? If yes, when the simulator is used as the dynamics model, shouldn't the solution found be very similar to percentage feasibility? Also, if CEM finds a solution in 3% of the cases, how is the success rate 40%? It would be really nice to explain this clearly.

summary_of_recommendation: I like the use of goal conditioned policy, task dynamics model and RRT style tree search to solve hard long horizon tasks. That's why I am giving it a weak accept. I will be happy to increase the score if authors are able to more clearly explain the metrics used in reporting the results (as pointed above) 
issues: Mentioned above. See the weakness section.

POST-REBUTTAL UPDATE: The authors have addressed my concerns. I am keeping my score to a weak accept. It's more of an accept for me (but the only other option available is strong accept)
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a task and motion planning algorithm that combines the classical motion planning algorithm, RRT, and reinforcement learning to solve long-horizon problems. The authors have shown its superior performance over baselines in simulation.
strengths_and_weaknesses: Strengths:
- The authors tackled an interesting problem that lies in the intersection of classical planning and end-to-end policy learning, which is gaining its popularity recently. I find the particular architecture of the proposed method novel. 
- Many figures are included to help better understand the paper.
- The results seem promising.

Weaknesses:
- The motivation of the work should be improved in comparison with task and motion planning algorithms in literature. Task and motion planning algorithms can also solve the problems in known, quasi-static environments efficiently so it is not clear why one should use the proposed complex method that involves with RL and user inputs. More emphasis on complex tasks or giving a concrete example that cannot be solved without RL would be helpful.
- Some parts are not technically sound. Detailed comments are included in Issues below.
- The paper is not well readable. It includes heavy terminologies, which might be improved by unifying similar ones, and figures require more information, such as notations, explanation on different lines and colors used, and what space it represents. The placement of figures, algorithm, and texts needs to be revised so that texts match with the right figure and algorithm. 
summary_of_recommendation: Although the paper proposes an interesting method in robot learning and planning, I find the paper hard to read and understand. Also, there are technically vague parts in the method that need to be clarified. Overall, the presentation needs an improvement. Especially, it is confusing how RRT interacts with RL setting, which is the most important part in the paper. Thus, I think the paper is on the borderline in its current form.
issues: - As mentioned in Weaknesses, the motivation should be enhanced in comparison with literature on task and motion planners.
- I think the paper addresses planning only, not interleaved planning and execution. However, the right figure in Fig. 2 and images of intermediate steps in Fig. 3(b) might make readers think that the method proposes an interleaved planning and execution approach. Please clarify this.
- The motivation of employing RRT is not clear. RRT is used to find a trajectory that does not collide with obstacles in the high-dimensional configuration space. Are all x points in configuration space and does the Fig. 4(a) represent the configuration space? In its current form, it looks like RRT is used to find a trajectory in obstacle-free state space, which can also be done with simpler planners.
- Fig. 3(a) needs a huge improvement since it represent the main method. It would be helpful to indicate which part is connected to RRT and include used notations. Also, since there are the task space and the state space, please indicate which space it is in all figures.
- The state is not an image but the state of the robot and object as explained in the experiments. From figures, readers might think that the bird-eye view images are used as a state. It would be good to clarify what state is considered.
- Along with the above comment, the problem description is not clearly stated before explaining the method. Please clearly state what is known and unknown to the robot, and what are inputs (e.g. what kind of data) to the proposed method.
- In the below texts in page 3, it would be helpful to explain the overall method using the line number of Algorithm 1 and Figure 3(a). Currently, texts, figure and algorithm are disconnected.
- The authors used the narrow passage as a fundamental issue in classical planning but it is only a fundamental problem of sampling-based motion planners. There are other algorithms in classical planning, such as trajectory optimization, that does not have the narrow passage problem so this is incorrect.
- The theoretical results do not seem to add much value but interrupts the flow of the reading. I would suggest to take it out.
- The considered task in experiments involves with pushing buttons. Is the state that the button being pushed or not represented by a state x? This is a discrete variable so how can RRT handle discrete variables with other continuous variables in search?
- I understand there is a page limit so that including all experimental results is hard. Although the validation on entire task is critical, readers might also be interested in seeing how each component performs. For example, is x_sample obtained by RRT for a given x_goal semantically meaningful?; is sampled z from task space also semantically meaningful with respect to neighboring points in task space?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors propose a planning method for using imitation-learned policies to form plans over multiple sequential tasks. They use imitation learning to train a goal-based policy, and train a temporal distance classifier to predict the time to reach the sampled point. The proposed planning algorithm is similar to performing RRT in a space of task manipulations. They sample points from the dataset of demonstrations and attempt to reach those points, expanding the lowest-cost node on the tree that they predict can reach the point in under a certain time. They find that when using a simulator or task-conditioned model, they are frequently able to complete difficult strings of tasks that are difficult for existing methods to solve. They also find that the task-conditioned model is essential for preventing the planner from exploiting inaccuracies in the model and maintaining accuracy over long time horizons. 
strengths_and_weaknesses: This method moves towards making it feasible to plan over sequences of tasks, which is a challenging problem. In particular, the complexity of the tasks that they attempt is impressive. I find it interesting as well that this is done without learning different networks for different tasks, instead learning by imitation without well-delineated 'tasks'. 



summary_of_recommendation: The experimental results for this method are strong, and planning method combined with the goal-conditioned policies make sense as a general problem-solving tool for complicated complicated tasks. The actual planning algorithm bears very strong similarity to RL-RRT, so the interest here is primarily on the technical details needed to achieve these kinds of results. Luckily, the method delivers these technical details and results. 
issues: It's not clear what the Chain-LMP method is or what it's doing. It's also not clear in what sense this method is an oracle, considering its low success rate. Another detail that I had difficulty making sense of was the difference between the success rate and the 'solution found' rate when using the simulator. It was my understanding that the experiments are performed on a simulator, so if the method finds a sequence of actions that solves the task with the simulator, why can't that sequence of actions just be directly applied to deliver a solution?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a framework that combines imitation learning and planning approaches to solve long-horizon control tasks. Specifically, a goal-conditioned model is trained via imitation learning to achieve short-horizon behaviors, and an RRT planner uses this goal-conditioned model to find a sequence of temporally abstract waypoints to solve long-horizon tasks. Experiments on a simulated playground environment demonstrate that the proposed approach can solve long-horizon tasks more effectively than a pure goal-conditioned baseline and a variant using an alternative planning method.
strengths_and_weaknesses: The overall goal of this paper — solving long-horizon tasks — is motivated well in the introduction, and the related work section sets the context for relevant work well. The first three figures — showing a progressively more concrete overview of the method — also complement the paper well. These components all help to understand the high-level ideas of the paper. That said, there are a number of areas for improvement regarding the clarity and technical quality of the methods and experiments section:
1. The definition of the goal-conditioned model is a bit unclear. The paper mentions that the underlying goal-conditioned policy is based on Play-LMP, which consists of both a goal state $g$ and a task vector $z$, yet in this paper the policy is only conditioned on $z$.
2. The metrics in the main experimental results (table 1) should be defined more clearly. It is possible to infer the distinction between “Solution Found” and “Success Rate” but these details should not be up to interpretation.
3. How long does it take the planner to find a solution? This is an important consideration if we seek reactive agents that can operate in real-time.
4. What is the dimensionality of the state space? This can go in the appendix.
5. Is the underlying Play-LMP policy trained on short-horizon windows of (state, goal) pairs or arbitrary horizons? If it is trained for short-horizon windows only then it makes sense why the LMP baseline performs poorly — in that case the LMP comparison would not be fair because the baseline was not trained to solve long-horizon tasks.
6. The LMP baseline is only evaluated on tasks that can be expressed as a single goal state, in contrast with the other baselines. It may be that single-goal state tasks entail dealing with a unique set of challenges, making this an unfair comparison.
7. Clarify the distinction between the play dataset and the demonstration dataset. Why is a separate demonstration dataset needed?
8. In the cases when the method does not find a solution, what are the underlying reasons behind these cases? Expand on the analysis.
9. It is entirely possible that the dynamics model predicts infeasible states, eg. when two blocks penetrate with each other. The success function may erroneously predict success for infeasible states, unless it can filter such states to start with. How does the success function deal with such states?
10. The paper claims that the proposed method can deal with high-dimensional states. Have you considered tasks operating over images? Are there specific modifications you need to make to your framework to deal with images?
summary_of_recommendation: This paper is well-motivated and the high-level idea of the paper is easy to understand. There are however a number of low-level details that are unclear in the current iteration of the paper. Refer to the “Issues” section for details. These issues are hopefully easy to address, **at which point I will raise my recommendation.**

---
Update: raised recommendation to weak accept, **but with some hesitation.** Please refer to my followup post for details.
issues: See the list in the “Strengths and Weaknesses” section.
review_title: null

Meta-review: ### Final Meta-Review

The authors have carefully responded to reviewers' questions and, after the revision and discussion process, reviewers agree that the updated manuscript has gained in clarity. The initial concerns have been mostly resolved and all reviewers now recommend paper acceptance. I echo their recommendation—nonetheless, the authors should note the reservations expressed by reviewer FQLy and address them to the extent possible in the final version of the paper.


### Original Meta-Review

The paper presents a hierarchical planning scheme combining task-level tree search with learned motion-level control policies. The reviewers agree the the paper is generally well motivated and introduces an interesting new approach. On the other hand, the reviewers express shared concerns regarding technical clarity of terminology and definitions as well as the evaluation metrics used; raised issues regarding the comparison to the state of the art and choice of baselines also need to be addressed.

Three of the four reviewers point out the lack of an explicit upfront discussion of the three metrics used to evaluate the performance of the method (solution found, success rate, and feasibility). As far as I can tell, "feasibility" is only defined in the caption of Table 1, but its meaning (somewhat surprising) has significant implications on gauging the method's effectiveness. I ask that the authors provide clear and formal definitions of these metrics so that the reviewers can fully assess the results.

One reviewer stresses that state-of-the-art task-motion planners are able to solve problems in this class successfully, and the proposed method needs to be more strongly motivated with respect to these existing approaches. The authors must respond to these comments, also explaining why their choice of experimental baselines did not include any methods in this family.

Two of the reviewers also raise questions regarding the state space and its representation (both continuous and discrete) in the proposed algorithm. The authors should respond to these questions, clarifying what representation is being used in each case (search tree, learned policy, paper figures, etc.). Similarly, please address the questions regarding goal and task representation.
Decision: Accept (Poster)