Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors propose an imitation learning method for long-horizon robot tasks that is train on task and motion planning (TAMP) data. They train a hierarchical policy, with a high-level policy that learns to predict abstract/symbolic actions and a low-level policy which has an attention component and generates low-level motions. The goal is to generate a sequence of actions that both (1) generates high-level symbolic plans, and (2) generates trajectories.

Training is done via a system inspired by Dagger where they run a task and motion planner in parallel alongside training. After training, they run plans and if they fail, collect additional training examples by invoking the expert TAMP system.

Training runs as a set of nodes: the "task planning" node runs something like FastForward to generate a sequence of high-level actions, a motion planning module which generates trajectories, and training. A motion planning node generates trajectories, a training node updates the model, and finally they perform structured exploration to collect more data (similar to Dagger) to do both. This is the big innovation the authors propose, which allows them to efficiently train this hierarchical policy on a variety of simulated tasks.

They perform experiments on 3 simulated domains, 2 of which are from RGB images.
strengths_and_weaknesses: Briefly:
+ Important problem
+ Reasonable performance on common benchmarks
+ Approach for training seems reasonable, and the distributed training architecture seems novel and helpful for large tasks where simulation is an option
- numerous obvious typos 
- state space often not clear to me 
+ Good comparisons to related work

I think the chief novelty is their distributed training architecture. The authors propose a system which shows performance on a range of different environments. They also show results both on RGB images and on joint state vectors and on a variety of problems and environments.

I have some concerns about how this system would scale, though. The authors performed some ablation experiments were performed; however, these weren't performed in the cases where the system was trained on RGB images.

The meaning of the object ablation experiments in RoboSuite seemed unclear to me; the authors should explain this in more detail and what it says. They do an analysis of different types of feedback, which is interesting. It seems as if tamp-feedback is not useful more than motion-feedback, which might imply the high-level task structure is too simple.

summary_of_recommendation: I think this paper tries to solve an important problem and brings some interesting ideas, and they get decent results, even if the results are not demonstrated in the real world and the generalization experiments they perform are overly limited. The ablation experiments focus on feedback types, not on holding out different goals.

The distributed training system seems like it could be very helpful, though, if it could be applied to training models that could generalize or adapt to real problems.

UPDATE: after the author's response I think it's more practical and stronger results than I had initially thought. Updated.
issues: 
My big concern with this method is that it seems extremely data intensive and not very practical. The system as a whole seems to work on environments and tasks that have been seen; one of the big advantages of TAMP systems though is generalization to new problems. 

How much data did your system end up needing? How many trajectories were collected? Some additional information would help determine how well it scales.

Ablation experiments holding out specific types of goals in RoboSuite would be very interesting and helpful.

Please make plots like Fig. 5 in color as well as using symbols; different shades of gray are very hard to parse.

Minor:
Abstract: let's --> lets
3.1:  A symbolic planner, such as Fast123 Fowardciteff, --> FastForward~\cite{...}
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a framework for learning hierarchical long-horizon policies using a task and motion planner in the loop. The framework is based on a flexible TAMP algorithm, which uses an explicit interface between task planning and motion planning. Using this decoupled approach allows performing each part independently and in parallel. The task planner uses a Fast Forward algorithm to find sequences of actions, while motion planning is performed using numerical optimization.

Since the output of the framework is a trained neural network policy, the TAMP plans are not necessarily carried out completely, rather they are used to generate datasets to train two policies (i.e., task and motion). These are in turn used to guide the search. 

Hence the authors propose feedback mechanisms between the hierarchical policies and the TAMP algorithm which provide the training dataset for the policies. These feedback loops allow lowering the sample complexity of the framework. The proposed approach is thoroughly tested using three different benchmarks using ablation as way to highlight the important part of the framework.

strengths_and_weaknesses: The paper is well written and presents the ideas and results in clear way. The main strength of the paper is twofold.

1)	The originality of the idea is very relevant and design choices are sound
2)	The choice experiments are interested and show the potential of the framework

Efficient motion policies for long-horizon manipulation tasks is clearly a very important milestone for robotics research. In this context very, few works combine learning and planning in an original way. As the authors point out most of the attempts use learning in the planning algorithms. In this work, the authors intertwine the two in such way that it minimizes sample complexity. For that using [15] seems very appropriate as it allows to decouple the task and motion planning problem.

Having experiments on three benchmarks allows to see, which parts of the framework are more important under which condition. For instance, considering feedback loops, the motion-feedback seems most important but sometimes task feedback can improve sample complexity in domains where the planner does not encounter the situations produced by the policy.

While these are two strong points of the paper. They could be further improved by considering the following: The paper mainly lacks a theoretical situation of the problem. 

While it seems difficult to analyze theoretically this work, the framing of the research is not completely accurate. The title suggests that the authors are solely imitating the TAMP policies and that they use feedback to increase sample complexity. While that may be true in practice, it seems that a more accurate framing of the work is terms of Reinforcement Learning. The work can be viewed as a hierarchical version of Guided Policy Search (GPS). This way it should be possible to provide a better theoretical derivation and experimental analysis of the work.

For instance, taking inspiration from GPS, one could study the trade-off for generating policies that suddenly diverge from what the TAMP algorithm produces. This could be enforced in the motion optimization component using a cost function as done in GPS rather than only from rollout initialization. A dynamics model could be learned for the motion optimization, a KL constraint could be added in the supervised training phase etc. 

Concerning the theory, deriving the approach as some kind of descent algorithm would provide more clarity even if the implementation approximates it. 

Finally, concerning the experiments framing the problem as a model-based hierarchical RL algorithm naturally yields comparison to other hierarchical RL approaches such as HIRO, FuN etc. which are absent from the paper.

summary_of_recommendation: It is recommended to frame this work in the context of Model-based hierarchical Reinforcement Learning, instead of active imitation learning. This reframing implies comparing to SOTA HRL algorithms such as HIRO or FuN. 
issues: See above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose a distributed training scheme for task and motion planning problems. This relies on access to symbolic robotics oriented task planner and motion planner to generate oracle training data for feed-forward policies that output task plans as action schemas and generate motion plans by chaining action schemas through reactive low-level control policies. The authors demonstrate the following:

1. Ammortizing planning by approximating TAMP solvers through hierarchical policy networks results in a high success rate for TAMP tasks.

2. Active exploration of problems space where the policy fails, and then generating oracle feedback can produce better quality data for optimization, by only using TAMP planner resources where feed-forward policies fail

3. Feedforward policies can speed up a planning based approach by generating better initial guesses.
strengths_and_weaknesses: **Strengths**

- The idea of ammortizing planning through approximation is an elegant way of retaining the flexibility of iterative planning, and online repair by narrowing the exploration space using good quality feed-forward policies. The authors idea of a distributed framework appears to be a powerful method to achieve this.

- The experiments are quite convincing towards the power of TAMP supervised feed-forward policies in comparison with RL-based or LfD-based policies. While this is to be expected as TAMP can provide a better training distribution, this work provides a convincing demonstration and a concrete modular implementation of this notion

**Methodological comments**

- The authors have restricted the action schemas to discrete outputs; the impact on problem domains that *require* continuous parameterizations in unclear. In theory,  the action selection network can output both discrete and continuous parameterizations, but the case of continuous parameters might be more challenging to train

- There are a lot of hand-engineered inputs provided to the problem (constraints, fluent descriptions etc.) that are not available to an RL-based baseline. While this is a representational advantage of using structured methods, this does place the burden of generating these on an automation engineer. 

- The authors' decision to create a separate policy network for each action schema can potentially cause issues with more number of objects, shared parameters to reduce the memory requirements for control policy architectures would be interesting to look at.

- The authors do not demonstrate how the policy networks performance is impacted by length of planning horizon required to solve the test problems. Does performance deteriorate when longer policies and task plans are required?

**Presentation comments**

- What problems are picked by the exploration nodes? Are they picked from a procedural problem distribution, a dataset of problems? The main paper does not address the role of the exploration nodes at all

- Task planners are limited by the size of the logical states, the authors should include the number of logical fluents for each problem domain. Further, the authors do not provide any elaboration of how the geometrical facts for non-feasible task plans are generated. 

- I would appreciate a longer discussion of how this work relates to prior approaches in learning to approximate TAMP solvers. I found this discussion quite superficial and focusing on the implementation details rather than the underlying aspects of TAMP solvers that are being learned by the models. In particular the approach by Driess et al. is closely related, and provides similar capabilities, but the authors' distributed approach can potentially provide faster scaling. The reader would appreciate a deeper comparison with prior work. The authors can potentially tighten the text in the results section or move some of it to the supplementary materials to provide a better framing of their approach. 
summary_of_recommendation: I believe that the idea of decentralized learning for approximating a TAMP solver is novel, yet the idea of approximating the TAMP solver by itself is not. I believe that the authors should frame their contribution as such

Further, the larger ideas are quite clear in the paper, but there are a lot of implementation details that are not elaborated enough.

I don't believe the work is purely incremental.

Therefore overall I recommend a weak accept, and I urge the authors to make some changes
issues: - Characterize the effect of planning horizon on the success rate. I would like to see how success rate deteriorates with longer required planning horizons

- A deeper discussion on differences between prior work on approximating TAMP solvers and how your approach builds on or is novel in that context.
review_title: null

Meta-review: This paper addresses the issue of jointly optimizing task and motion planning (TAMP), and proposed a method which includes to imitate a TAMP solver with a trained policy and to fast update the TAMP solver with a hierarchical policy architecture. 

I agree with reviewers that this paper is well organized and the idea is quite novel, the experimental results illustrated this sutdy is effective and promising. I believe this is a good work.

I also agree with reviewers on some minor points they mentioned, and suggest authors carefully address these comments to further improve this paper.

More added:
I would like to thank authors for your carefully and detailed responses to comments raised by reviewers. Authors responses seems to be effective, since the score of this paper was improved after rebuttle.
Also thanks feedbacks from reviewers.
I really believe the revised version is more better. 
Congratulations!

Decision: Accept (Oral)