Guided Imitation of Task and Motion Planning
Authors: Michael James McDonald
Dylan Hadfield-Menell
Venue: CoRL2021 Oral (2021)
Keywords: task and motion planning
mobile manipulation
imitation learning

Abstract: While modern policy optimization methods can do complex manipulation from sensory data, they struggle on problems with extended time horizons and multiple sub-goals. On the other hand, task and motion planning (TAMP) methods scale to long horizons but they are computationally expensive and need to precisely track world state. We propose a method that draws on the strength of both methods: we train a policy to imitate a TAMP solver's output. This produces a feed-forward policy that can accomplish multi-step tasks from sensory data. First, we build an asynchronous distributed TAMP solver that can produce supervision data fast enough for imitation learning. Then, we propose a hierarchical policy architecture that lets us use partially trained control policies to speed up the TAMP solver. In robotic manipulation tasks with 7-DoF joint control, the partially trained policies reduce the time needed for planning by a factor of up to 2.6. Among these tasks, we can learn a policy that solves the RoboSuite 4-object pick-place task 88% of the time from object pose observations and a policy that solves the RoboDesk 9-goal benchmark 79% of the time from RGB images (averaged across the 9 disparate tasks).

Guided Imitation of Task and Motion Planning
Michael J. McDonald University of California, Berkeley m.j m.j.mcdonalddee beerkely..cddu
Dylan Hadfield-Menell Massachusetts Institute of Technology dhm dhm@csail.min cstii..mit.duu
Abstract: While modern policy optimization methods can do complex manipu- lation from sensory data, they struggle on problems with extended time horizons and multiple sub-goals. On the other hand, task and motion planning (TAMP) methods scale long horizons but they are computationally expensive and need to precisely track world state. We propose a method that draws on the strength of both methods: we train policy imitate a TAMP solver's output. This pro- duces feed-forve fordard policy that can accomplish multi-step tasks from sensory data. First, we build an asynchronous distributed TAMP solver that can produce supervision data fast enough for imitation learning. Then, we propose a hierarchi- cal policy architecture that lets us use partially trained control policies to speed up the TAMP solver. In robotic manipulation tasks with 7-DoF joint control, the par- tially trained policies reduce the time needed for planning by factor of up to 2.6. Among these tasks, place task 88% of the time from object pose observations and a policy that solves the RoboDesk 9-goal benchmark 79% of the time from RGB images (averaged across the disparate tasks).
learn policy that solves the RoboSuite 4-object pick-
Introduction
This paper describes a policy learning approach that leverages task-and-motion planning (TAMP) to train robot manipulation policies for long-horizon tasks. Modern policy learning techniques can solve robotic control tasks from complex sensory input [1, 2, 3]. but that success has largely been tasks. remains an open problem to learn policies that execute long se- quences of manipulation actions [4, 5]. By contrast, TAMP methods readily solve problems that require dozens of abstract actions and satisfy complex geometric constraints high-dimensional configuration spaces [6, 7]. However, their application is often limited to controlled settings be- cause TAMP methods need to robustly track world state and budget time for planning [8, 9].
limited to
To address these concerns, two lines of work have emerged. The first uses machine learning to identify policies or heuristies that reduce planning time [10, 11, 12, 13, 14, 15]. This extends the domains where TAMP can be applied, but still relies on explicit state estimation. The second line of work learns policies that imitate the output of TAMP solvers. These approaches reduce dependence on state estimation by, e.g., learning predictors to ground the logical state [16] or training recurrent models to predict the feasibility of a task plan [17]. These learned policies operate directly on per- ceptual data, and do not need hand-coded state estimation. However, TAMP solvers define highly non-linear mapping from observations to controls and this makes imitation difficult. Furthermore, the compute resources required for planning limits the availability of training data for learning.
In this paper, we build on both approaches. We take inspiration from guided expert imitation [18, 19] and train feed-forward policies to imitate TAMP. Our key insight is that Srivastava et al. [20]'s modu- lar TAMP framework supports distributed TAMP architecture that: 1) is optimized for throughput (as opposed to low latency, the typical focus in TAMP research); 2) trains policies to imitate the planner output; and 3) uses those policies to amortize planning across problems and reduce compute costs. The result is virtuous cycle where learning accelerates planning, and faster planning pro- vides more supervision. We propose a policy architecture that leverages the definition of the TAMP domain to improve learning performance. Task plans from our system supervise task-level model that learns to predict parameterized action. This output selects from and parameterizes set of control networks, one for each action schema. We show that this architecture can execute manipu-
5th Conference on Robot Learning (CoRI. 2021), London, UK

---

kt
goal
H---D
grip
controll poliicess
Alwes
attention
Plastios
10
Figure 1: Left: We train hierarchical policies imitate the output of distributed task and motion planning system. The policy has two components, Thi and Wio Thi takes observations a goal and produces one-hot encoding of the choice of abstract action and associated parameters. These outputs and the original observa- tion pass to the motion-level policy Th consists of two stages: attention module and action-specific control networks. The attention module maps the continuous observation and discrete action parameters to continuous parameterization. There one control network per action type and the choice of abstract action gates the outputs of these controllers to produce the next control. We denote this combined policy as Tio o This Right: We evaluate in two simulated robotics domains: RoboSuite [21] (top) and RoboDesk [22] (bottom). In RoboSuite we train policies from object poses that reach 88% success on the four object variant of the domain. In RoboDesk, we train policies from RGB image data and reach 79% success on the 9-goal multitask problem.
lation skills in nigh-dimensional environments from complex sensory input. As result, system able to compile task and motion planning into a single feed-forward policy.
The contributions of this work are follows: 1) we show design for distributed, asynchronous, high-throughput task and motion planning system that leverages policy learning to speed up plan- ning: 2) we propose hierarchical policy architecture that leverages the TAMP problem specifica- tion; and 3) we implement and evaluate this method to show it can learn to accomplish multiple goals over (comparatively) long horizons with nigh-dimensional sensory data and action spaces. In a 2D pick-place task, we train policies that place objects precisely onto targets 83% the time from RGB images. In the RoboSuite [21] pick-place benchmark train policies for 7-DoF joint control that place 4 objects 88% of the time from object pose vectors. In the multitask RoboDesk benchmark, learned policy averages 79% success rate across 9 diverse tasks with 7-DoF joint control from RGB images.
2 Background
Task and Motion Planning. Task and motion planning (TAMP) divides a robot control problem into two components: symbolic representation of actions (e.g., grosp) and a geometric encoding of the world. Task planning operates on logical representation of the world. It finds sequences of abstract actions to accomplish goal (e.g.. pick(obji). place(obj1. targı).... ...). Each action encodes motion problem that must be refined (i.e., solved) to obtain feasible trajectory that satisfies specified constraints.
We represent TAMP problems with the formalism introduced by Hadfield-Menell et al. [23]. A
TAMP problem tuple (X, F,G. G, J, A). X is the space of valid world configurations. F is F,,,G. a set of fluents, binary functions of the world state that characterize the task space ʃ : X - {0. 1} (c.g. at(obja, targ2) or holding(obji)). G is the goal state, defined U is the control space of the robot. f describes the world dynamics: f(z,, u) = It+1- xo describes the set of abstract action schemas. Each action
conjunction of fluents {..).
the initial world configuration To E X. Finally, schema a has four components: 1) a.params: the parameters of the action (e.g., which object to grasp); 2) a.pre: set of gurameter-dependee
fluents that defines the states when this action can be taken; 3) a.mád: set of fluents that constrains the allowable controls for this action: and 4) a.post: a set of fluents that will true after the action executed. Solutions to such a problem are a pair of sequences (α, 7). where ă encodes abstract actions and 7 encodes refined motion trajectories. A plan is valid if 1) the initial state of each т; satisfies a,-pre; 2) each t; satisfies a;,mid; 3) the end state of each satisfies a; post; and 4) the final state satisfies the fluents that define the goal.

---

symbolic plan
nodes
Motion Planners
refinement result
Problem Distribution
Task Planners
Fl failure
initial trajectory
nodes
Wio
Exploration
successfully refined plan
successful
trajectory
training
Te training L(
nodes
Figure 2: The system contains five core components which run in parallel and communicate through shared data structures. This division allows for the execution of arbitrary copies of motion planning, task planning. and policy rollouts and can scale to utilize available hardware.
Modular TAMP. Our approach parallelizes the modular TAMP approach of Srivastava et al. [20]. They use graph representation of TAMP, where nodes correspond to different task plans. A node can be either refined expanded. Refinement uses motion planning to search for trajectories that accomplish the actions in the plan. Refinement failures are tracked along with the unsatisfiable constraints that caused the failure. When a node is expanded, one of these refinement failures is used to generate new plan. The error information (e.g., collision with obstacle) is used to update the abstract domain so that the planner can identify an alternative plan (e.g., one that moves the obstruction out of the way). To solve TAMP problem, Srivastava et al. [20] interleave refinement and expansion to find a plan that reaches the goal.
To implement plan refinement, we use the sequential convex optimization method described in Hadfield-Menes et al. [23]. This formulates motion planning as an optimization problem. The objective is smoothing cost || E.llT++ - Tall The constraints are determined by the pre, post, and mid conditions of an action schema: To E a.pre, Tr E a.post, and a.mid. E.g. for grasp action the preconditions constrain the initial state to position the gripper near the object with proper orientation. The postconditions constrain the final state so that the object is in a valid grasp. The midconditions constrain the intermediate states to avoid collisions. If this optimization fails, we use the unsatisfied constraints to expand the associated plan node, as described above.
Method
This section has three parts. First, we propose a distributed TAMP solver. Then, we describe a hierarchical policy architecture that leverages the TAMP domain description make it easier to model TAMP behavior. Finally, we show how to incorporate feedback from the learned policies to prevent trajectory drift.
3.1 Distributed Planning and Training Architecture
Figure 2 shows our overall design. Our system has four types of nodes that operate asynchronously: 1) policy training; 2) task planning; 3) motion planning: and 4) structured exploration. The task-level policy тhi and the motion-level policy Tio train within their respective nodes without backpropaga- tion between the two. For this work we found standard supervised learning sufficient for good complex pro- cedures (e.g. generative adversarial imitation learning [24] or inverse reinforcement learning [25]). Specific network architectures and hyperparameters are included in the Supplemental Materials.
performance. However the modular design makes it straightforward to apply
Algorithm outlines our task planning procedure. reads from a shared priority queue Qtask that tracks task planning problems, represented as tuples of an initial state a logical state Pu. goal g. and a refined trajectory prefix Tp. At random intervals, or when the queue is empty, we sample new planning problem from the base problem distribution, with empty trajectory prefix to 0. symbolic planner, such as FastFoward [26], computes valid action sequence ă that achieves and pushes the problem (ro, Po. g, 7o) and action sequence to the motion queue Q Qootion-
3

---

Algorithm 2 outlines our motion planning procedure. It pulls from Qmotion and applies the refine- ment procedure from section to compute valid trajectory segment T' for each a'. If refinement fails, it computes the set of unsatisfiable constraints {es} that describe why the motion planner could not refine a', appends these to the original logical description Φg. and pushes the updated problem instance (zo, To J [ei].g, ,0:-=1) to Qtakk- If refinement succeeds, each 7f is pushed to the motion dataset Dootio to supervise the appropriate control policy. When the goal is reached (i.e., TT E 9), it pushes the action sequence ă to the task dataset Drask to supervise the task-level policy.
3.2 Hierarchical Task and Motion Policies
Learning to imitate TAMP solutions with single policy is difficult because small changes in the observation can lead to different task plans and, thus, radically different controls. We deal with this through a hierarchical policy architecture that mirrors the TAMP domain description. The policy is split into two parts: task-level policy Thi to select parameterized actions a, and motion-level policy To select controls ur as a function of a, and world state. Figure illustrates our design.
The maps continuous observations and goals to the discrete space of parameterized action schemas. We use the factored encoding scheme from Van al. [12]. Thi outputs sequence of vectors. The first specifies a one-hot encoding of the choice of action-type (e.g. place). Each subsequent vector provides a one-hot encoding of the ordered action parameters (e.g. objl, targ2). Training data for this policy is pulled from the task dataset Drosk-
Algorithm Task Planning Node
Require: Shared queues Qtask, Qmotion Require: Problem distribution P
1: while not terminated do if imppetty((task) then (zo. 2,g, T) ~ Pproh
2:
3:
4: else
5: (Zo. U.g. +- pop(Qtask)
6:
ă + ttask-ppan 2 g)
7:
push(Q motion (xo. 4 g. a))
The low-level policy Ti consists of two stages:
an attention module and action-specific control networks. The attention module maps the con- tinuous observation and discrete action parame- ters to a continuous parameterization. This step flexible. In our system, it represents the rel- evant objects in a particular geometric frame. E.g., for the action place(obj targ2) this is the pose of obj'l in the targ2 frame. When the policy has access to state data, we hard code this step. In other situations (e.g., learning from camera images), this is learned from su- pervision alongside the control policy. The ab- stract action, the continuous parameterization, and the original observation pass to the second stage to predict the next control. This stage con-
Algorithm 2 Motion Planning Node
Require: Shared queues Qmotions Qiask Require: Expert datasets D Dmotion Drask Require: Motion policy Tmotion
1: while not terminated do
2:
(x, 4. g, T Tā)) +- Qmotion for a; E ā do
3:
4:
T² + rrollou((zr, Tio(+
T'. success +- motion plan (Ir diy f+)
5:
6:
if success then
7:
+-1--]
8:
append(r, 7')
9:
push( motion (a., 7'))
10:
else
tains a set of separate control networks, one per
11:
## add unsatisfiable constraints {e:}
action schema. Training data for this policy is
12:
push(Qrask: (I, & U (c:}.g. g..))
pulled from the motion dataset D motion-
13:
break
14:
if r[-[ E g then push(Drask: (T, ā, g))
3.3 Policy-Aware Supervision
A common challenge imitation learning is that small deviations from training supervision build up over time [27]. To account for this, we propose task and motion supervision methods based on Dataset Aggregation [28]. For controls, we bias trajectory optimization to provide supervision near states encountered by the motion policies. After an initial training period, we warm-start opti- mization with a rollout from the appropriate motion policy. Then, to keep supervision close to this trajectory, we modify the optimization objective with a rollout deviation cost, based on the accelera- tion kernel from Dragan et al. [29]. This uses a generalization of Dynamic Movement Primitives [30] that adapts the rollout trajectory to satisfy optimization constraints. For a sampled trajectory ff, the optimization cost is | - 111 E - 7²) - (Tf - +))l.. This is conceptually similar to the regularization penalty from Guided Policy Search [18].

---

2D Pick-Place
RoboSuite Pick-Placce
RoboDesk
0.8
Soccess Raato
0.6
0.6
0.6
0.4
0.4
nigeet (BG0))
Deval-and-ailkk
0.2
wetrip
obgjct
alboaley
Shali, Dour Closed (RGB))
0.0
0.0
0.0
Train Time (hours)
Figure 3: Average success rate over the course of training. Left: 2D pick-place domain. Training from RGB images and LIDAR-style sensors, the system learned place 2, and objects 99%, 97%, and 83% of the time, respectively. Observing object positions, learned to place 5 objects 88% of the time. Center: RoboSuite pick-place domain. Training from joint angles with object positions and orientations, the system learned to place 1, 2, and 4 objects 97%, 90%, and 88% of the time, respectively. With Gaussian observation noise it could place 4 objects 64% of the time. Right: RoboDesk domain. Training from joint angles and RGB images, the system learned place the block in the shelf and close the door 89% of the time. Across the multitask 9-goal benchmark, the learned policy averaged a success rate of 68%, or 79% with an added gripper camera.
To generate task-level supervision, the exploration node identifies states where the task policy gen- erates invalid actions. We sample problem from the problem distribution and roll out the combined policy with an execution monitor that ensures that high-level action is only started when a.p are satisfied and is executed until a.post are satisfied. We use this to build up a dataset of negative exam- ples for the task policy and to identify states where TAMP supervision may improve performance. Details and pseudo-code are in the Supplemental Materials.
Experimental Results
We evaluate our system in three domains: 2D pick-place simulator, the RoboSuite benchmark [21], and the RoboDesk benchmark [22]. Screenshots of the simulation environments are shown in Fig- ure 1. We used FastForward [26] the task planner. Unless otherwise stated, evaluations used processes for task planning, 18 for motion planning, and, when applicable, 10 for supervised explo- ration. Experiments ran for random seeds unless otherwise noted. Policy architecture details and hyperparameters are in the Supplemental Materials.
4.1 Learning 2D Pick-Place
Setup. We begin with simple, synthetic, pick-place domain. The robot is circular with par- allel jaw gripper and its goal to transfer up objects each to one of randomly assigned targets. The domain has three action schemas: noveto-and-grasp(c and-retreat(ob)) targ). The full space includes a grounding of these for each object and target. With 5 objects and 8 targets, this gives 85 possible actions. The control space outputs y, and rotational velocities on the robot body and an open/close gripper signal. We used DMCon- trol [31] for the underlying physics engine. The goal requires each object overlap with its target (i.e. [ s prs ll2 I radiusotj). We use simulated LIDAR sensor with 39 distance readings to facilitate collision avoidance.
transfer(obj, targ), place-
Manipulated Variables and Dependent Measures. Our primary variable is the training procedure used. We compare against: 1) flat imitation learning (flat-IL.,, single feedforward network trained to imitate TAMP output; 2) reinforcement learning with a dense reward function, trained with double the compute of our method (60 vCPUs); and 3) passive imitation learning (no-fcedback), which uses the hierarchical architecture but turns off policy-aware supervision (20 vCPUs). We used two RL methods: PPO [32], flat RL method, and HIRO [33], hierarchical method. See the Supplemental Materials for training details. We used two performance measures: 1) success rate (Suce.), the rate at which trained policies achieve the goal; and 2) distance reduced (Dist), the percent reduction in the distance from each object to its target.
Results. Table I shows the results of these comparisons. With I object, we observed 12% success rate for PPO and 1% success rate for HIRO. Flat-II. succeeded 21% of the time in the object variant. With two objects, it reliably reduced distance to the goal, but only succeeded 1.5% of the
5

---

Supervision Acceleration
RoboSuite Object Ablation
RobeSuite Pick-Place
Traj. Opt. Speecupp Ratio
2.4
0.0
0.8
1.8
0.6
0.6
1.2
0.4
0.4
feedbbekk
can only cecoul crrol--add-aade
0.6
PalkbDok
0.2
lask feedback
snolises fexllbckk
0.2
SuboSSule
0.0
1000
2000
1000
2000
Train Time (hours)
Total # Refined Plans
Figure 4: Our architecture utilizes partially trained motion policies to speedup plan refinement and improve data quality. Left: Speedup of plan refinement. During training, our method gave 2x speedup of plan refine- ment in the RoboSuite domain with two objects and 2.6x speedup in the RoboDesk domain. Center: Ablation of performance as a function of supervision trajectories (Total Refined Plans) for the RoboSuite domain with two objects. Right: Performance as a function of supervision trajectories across RoboSuite domains.
time. With 5 objects, we observed an 80% success rate for the hierarchical policy without feedback. This highlights the strong inductive bias our hierarchical architecture provides. With feedback, we observed success rates 99.7%, 98.7% 7,, and 88% for 1, 2, and objects. Performance plateaued on the 5-object variant with approximately 3.6 x 10 environment transitions of supervision, generated by 1600 successful plans over 2 hours. See Figure 3 (left) for depiction of performance over the course of training.
Camera Observations. We investigated the ability of our approach to train policies from RGB data. Figure: 3 shows perfor- mance over time for solving problems with 2, and objects. After hours of train- ing (120 cpu hours) our policy success- fully solves the object task 83% of the time. In all cases, performance plateaued with less than 3.4 x 10t timesteps of su- pervision from the planner.
obj
obj
5 obj
3HL/ / 29TS
6HL/ / 85TS
30HL/ / 223TS
Method
Suce.
Dist.
Suce.
Dist.
Suce.
Dist.
PPO
12%
33%
0%
10%
IV/
n/a
HIRO
1%
13%
n/a
n/a
n/a
a/a
Ilat-IL
21%
55%
1.5%
51%
n/a
m/a
n/a
n/a
n/a
n/a
80%
97%
tamp-feedback
99.7%
99%
98.7%
99%
88%
97%
Table 1: Comparison of average success rate (Suce.) and distance towards goal (Dist.) at completion of training for different learning methods on state data the Pick-Place domain. Dist. denotes the percentage reduction in the dis- tance from cach object to its target. HL denotes the number of high-level actions the planner would use solve the prob- lem and TS denotes the number of environment transitions induced by those actions.
Generalization. We tested the learned policies' ability to generalize to novel goals and dynamics at test time. First, we introduced 6 additional obstacles to repre- sent 'humans' in the environment. The policy observations were the same as before, so the robot had to use the LIDAR sensor to avoid the humans. At training time the humans stayed in fixed, random locations. At test time, the humans use model predictive control to take actions that move towards random goal location. Their objective penalized collisions with other humans, but not the robot. Rollouts terminated if human collided with the robot. We observed a 50% success rate placing objects in this condition. Next, we moved the target locations for objects from training to test time. We arranged the targets in a square at the center of the free space at training time. At test time, we moved the target locations to the edges of the space. This placed all targets outside the region seen during training. We observed success rates of 99%. 97%, and 39% for problems with 1, 2, and objects, respectively.
4.2 Learning Pick-Place for 7-DoF Robotic Arm Control
Setup. We measured performance in the RoboSuite pick-place benchmark with a simulated 7-DoF Sawyer arm [21]. The goal is move a cereal box, milk carton, soda can, and loaf of bread from random initial locations in bin on the left to their target bins on the right. The world state is the joint configuration and the positions of the objects. The action space specifies velocities for the 7 joints and a binary open/close signal for the gripper. The symbolic domain has 16 high-level actions across 4 action schemas: moveto (obj), moveholding The policy observations were joint angles, gripper state, the translation and rotational displacements from the gripper to each object, and the displacement from each object to its target. Our main experiment had 4 conditions: 1) can-only, to compare with prior work; 2) cereal-and -niimik,,, the most
and putdown(obi)

---

difficult 2 object condition; 3) all, the standard benchmark goal; and 4) all-noisy, which added Gaussian observation noise the policy observations with standard deviation equal to 1% of the possible deviation in initial values. We performed an ablation study in the cereal-and-milk condition to compare performance across feedback types: 1) no-feedback; 2) task-feedback (only); 3) motion- feedback (only); and 4) tamp-feedhack (i.e., both feedback types).
Dependent Measures. To compare with published results in RL, we report the average dense return of the trained policies over 500-step horizons for can-only. Our primary measure of performance success rate. We also measured the number of timesteps taken to reach the goal on successful trajectories. For the ablation study, we tracked the amount of time spent during motion optimization calls measure the speedup from feedback.
First, we compare against published results from RL and LfD. The official benchmark reports returns near 50 when using Soft-Actor Critic move the can [21, 34]. In comparison, we observed an average return of 324 for this task. Fan et al. [35] report an aver- age return below 600 over a 2000-step horizon. Our return for this horizon at least 1734. We compare with LfD results on success rate. Mandlekar et al. [36] trained from the RoboTurk cans dataset [37] and observed success rates of 31% from state data and 43% from RGB camera images. In comparison, we observed a 98% success rate from state data for our can-only condition.
Comparison to Prior Work.
Figure 3 (center) shows success rates time for the four conditions described above. Our learned policies reach an 88% success rate in the all condition, 90% in the sereal-and-min condition, and 98% in can-only condition. These correspond to average (successful) rollout lengths of 244, 120, and 55, respectively. In the all-noisy condition, we observed a success rate of 64%, compared with a 15% success rate executing plans from the TAMP solver directly. Figure 4 (center) shows success rate against training time for the ablation study. Both motion-feedbace and tamp- feedback reach 80% success within 2 hours. Task-feedback shows steady improvement but does not reach the performance of the other variants within the time limit. (See the Supplemental Materials for non-stationary condition where task-feedback improves on motion-feedback. The no-feedback condition does not get above 20% success. We observe up to a 2x speedup of the motion optimization code with tamp-feedbeel compared no-feedback. Figure 4 (left) illustrates the results.
Results.
Generalization. We ran a followup experiment to test the ability of the learned policies to generalize to unseen goals and dynamics. First, we held out goals that moved 3 and 4 objects and tested with other sizes of goals. When trained on problems that move 1 and 2 objects, the system was able to solve 3 object problems 48% of the time. When trained on 1, 2, and 4 object goals, it able to solve 3 object problems 58% of the time. Next, we trained on 1, 2, and 3 object problems and observed 60% success rate when tested with a goal to move all 4 objects. Note that this condition, which learns a multitask policy, is harder than the RoboSuite benchmark, which only attempts to learn single-task policy.
Next, we used the domain randomization functionality of RoboSuite to test the ability of the pre- viously trained policies to generalize to different dynamies. This varies properties of the physics engine such as the inertia and mass of the robot and objects, the parameters of the solver for con- tact forces, position and quaternion offsets of bodies, and physical properties of the joints. Using the default randomization parameters, we observed a success rate of 34%. When randomizing all parameters except the positional and quaternion parameters, observed a 73% success rate.
4.3 Multitask Learning for 7-DoF Robotic Arm Control from RGB Images
Background. We completed the core of this work on May 19th, with the above domains [38]. Our final experiment anecdotally measures the enginecring effort required to adapt the system to previously unseen environment. The RoboDesk multitask benchmark had been released 6 days prior, on May 13th [22]. Our final results reflect one engineer's 's effort over the course of three weeks to train on the benchmark, as well as two further weeks to train custom variants of the benchmark. The primary engineering steps were: 1) extension of existing action schemas to handle new tasks; 2) integration of the existing system with the new simulator; and 3) iteration on motion specification and ML hyperparameters.
Setup. RoboDesk provides a 7-DoF PANDA arm with 9 disparate tasks: open slide, press button, lift block, open drawer, block in bin, block off table, lift ball, stack blocks, and block shelf. We added composite goal: block in shelf with door closed. Our policy observations contain only RGB
7

---

camera images and joint positions. The controls are on joint velocities and the gripper. Our domain has action schemas: moveto(ob)), lift(obj),, stack obj2), place(obj. targ). press(button), open(door), close(door). This gives 31 possible high-level actions.
Nvidia DGX Station (40 vCPUs, 4 GPUS). On the composite task, the policy had a success rate of 89%. On the benchmark with the camera, the learned policy had a success rate of 68% averaged across the lasks. That rose to 79% when we added a second camera attached to the robot gripper. We report the benchmark performance for each goal in the Supplemental Material. In this domain, we observed a 2.6x reduction in motion optimization time from feedback as shown in Figure 4 (left).
Results. Figure 3 shows training results over 10 hours
default forward-facine
5 Related Work
Speeding up TAMP. There a growing field of work that uses machine learning on previous ex- perience to reduce planning time for TAMP systems. One focus is on guiding the task-level search something we do not explicitly address in our system. Chitnis et al. [15] trained linear heuristics from expert demonstrations. Kim et al. [11] used score-space representation to guide the search by transferring knowledge from previous plans. Wells et al. [10] learned a classifier to assess motion feasibility and incorporated it heuristic into the search. At the motion level, Ichnowski et al. [13] also learned policies to warm-start trajectory optimization. The key differences are that they predicted full trajectories, rather than executing the policy, and they did not penalize optimization for deviating from the policy controls. We note that our focus is on the learned policies themselves, and so we do not directly compare the speedup produced by our system other methods.
Imitating TAMP. Several other methods distill all or part of TAMP into learned policies. Paxton et al. [14] used imitation learning to train policies in an abstract task space and deep Q-learning in a control space. Like our system, they learned feed-forward policies for both task and motion prediction and used those policies to guide supervision. The key difference is the type of planner used. Ours is more compute intensive, but scales to larger problems. Kase et al. [16] proposed a method to imitate hand-engineered trajectories to train model to predict logical state and controllers to execute actions. Their method does not directly predict tasks, but allows a TAMP system to run online. In contrast, train single policy that maps observations directly controls.
The approach most similar to ours is that of Driess et al. [17]. They also train hierarchical policies to match TAMP output. They show that their system is able to integrate geometric and high-level reasoning for reaching task that, depending on the position of the target, potentially requires the use of tools. The primary difference is that their approach predicts the feasibility of a given action sequence, based on the initial state, while we predict actions and controls given the current state. This means that they need to search over candidate high-level action sequences at test time. In our system, this search is handled by the task planning node during training and bypassed entirely at test time. The other large difference is that they represent their control policies with learned energy functions that are minimized online. Task transitions occur when this system reaches an equilibrium state. In contrast, we learn policies that directly choose when to transition between tasks.
6 Future Work
First, this system should be deployed on physical robots. We hope that our policies will transfer more readily than those produced by, e.g., RL because TAMP solutions do not rely on detailed dynamics models. Next, it would be interesting to experiment with parameter sharing for the motion policies to save space and potentially speed up training. It would also be interesting to experiment with the output space of the high-level policy to directly output continuous parameters, which may allow policies to generalize more effectively. There is likely room to improve on both the task and motion policy learning: sequence modeling methods from NLP [39] may be better suited to represent task plans and modern imitation learning procedures, such as GAIL [24] or related works, may reduce training time or improve generalization. Note that each of these changes is relatively straightforward to implement, as it only modifies a single node of the training architecture. Finally, it would be interesting to identify theoretical guarantees on the learned policies. For example, it may be possible to formalize this as optimizing policy parameters with respect to a mixed discrete- continuous cost function defined by the TAMP domain.
8

---

7 Acknowledgements
We wish to express our deepest appreciation to Professor Anca Dragan for her support and advice throughout the course of this work, and for providing the resources needed to see it through to com- pletion. We would also like to deeply thank Thanard Kurutach for providing his advice throughout the project. We are furthermore grateful to our anonymous reviewers from the 2021 Conference on Robot Learning, whose feedback led to significant improvements both to the text and experi- mental results of this paper. This work was supported in part through funding from the Center for Human- Jomaa-Compatiblee Al.
References
[1] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-40,, 2016.
[2] J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. Trust region policy optimiza- tion, 2017.
[3] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L.. Weng, Q. Yuan, W. Zaremba, and Zhang. Solving rubik's cube with robot hand, 2019.
[4] T. Yu, P. Abbeel, S. Levine, and C. Finn. One-shot hierarchical imitation learning of compound visuomotor tasks. ArXiv, abs/18 18..811.111111111 1003, 2018.
[5] A. Li, C. Florensa, I. Clavera, and P. Abbeel. Sub-policy adaptation for hierarchical reinforce- ment learning. ArXiv, abs/1906.05862, 2020.
[6] L. P. Kaelbling and T. Lozano-Péres Hierarchical task and motion planning in the now. In 2011 IEEE International Conference Robotics and Automation, pages 1470-1477, 2011.
[7] T. Ren, G. Chalvatzaki, and J. Peters. Extended task and motion planning of long-horizon robot manipulation. ArXiv, abs/2103.055555 2021.
[8] C. Garrett, R. Chitnis, Holladay, B. Kim, T. Silver, P. Kaelbling, and Lozano-Pérer Integrated task and motion planning, 2020.
[9] G. Kazhoyan, S. Stelter, K. Kenfack, S. Koralewski, and M. Beetz. The robot household marathon experiment. ArXiv, 0oos/2011.09790 2020.
[10] A. Wells, N. Dantam, A. Shrivastava, and L. Kavraki. Learning feasibility for task and motion planning in tabletop environments. IEEE Robotics and Automation Letters, 1P:1-1, 01 2019.
[11] B. Kim, L. P. Kaclbling, and T. Lozano-Pérez. Learning guide task and motion planning using score-space representation. 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2810-2817, 2017.
[12] H. M. Van, O. S. Oguz, Z. Zhou, and M. Toussaint. Guided sequential manipulation planning using a hierarchical policy. RSS Workshop on Learning (in) Task and Motion Planning, 2020.
[13] Ichnowski, Y. Avigal, V. Satish, and K. Goldberg, Deep learning can accelerate grasp- optimized motion planning. Science Robotics, 5(48), 2020.
[14] C. Paxton, V. Raman, G. D. Hager, and M. Kobilarov. Combining neural networks and tree search for task and motion planning in challenging environments. CoRR, 2017.
[15] R. Chitnis, D. Hadfield Hadfield-Menel, A. Gupta. S. Srivastava, E. Groshev, C. Lin, and P. Abbeel. Guided search for task and motion plans using learned heuristics. In 2016 IEEE International Conference Robotics and Automation (ICRA), pages 447-454, IEEE, 2016.
[16] K. Kase, C. Paxton, H. Mazhar, T. Ogata, and Fox. Transferable task execution from pixels through deep planning domain learning, 2020.

---

[17] D. Driess, J.-S. Ha, R. Tedrake, and M. Toussaint. Learning geometric reasoning and control for Long-horizon tasks from visual input. 2021.
[18] Levine and V. Koltun. Guided policy search. S. Dasgupta and D. McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Pro- ceedings of Machine Learning Research, pages 1 1-9, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[19] E. Groshev, A. Tamar, S. Srivastava, and P. Abbeel. Learning generalized reactive policies using deep neural networks. CoRR, 0e5//708.072888 2017.
[20] S. Srivastava, E. Fang, L. Riano, R. Chitnis, S. Russell, and P. Abbeel. Combined task and motion planning through an extensible planner-independent interface layer. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 639-646, 2014.
[21] Y. Zhu, J. Wong, A. Mandlekar, and R. Martin-Martín robosuite: A modular simulation framework and benchmark for robot learning. 2020.
[22] H. Kannan, D. Hafner, C. Finn, and D. Erhan. Robodesk: A multi-task reinforcement learning benchmark. https://gi httpt.tthhhh..eee www. www..///ooooeeeeeeeeceeeeeeeee/e reaellerrreaarccrooooeeeee 2021.
[23] D. Hedfield-Menell, C. Lin, R. Chitnis, S. Russell, and Abbeel. Sequential quadratic pro- gramming for task plan optimization. In 2016 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pages 5040-5047. IEEE, 2016.
[24] Ho and S. Ermon. Generative adversarial imitation learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[25] Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-First International Conference Machine Learning, ICML '04, page 1, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581 138385.
[26] J. Hoffmann. Ff: The fast-forward planning system. Al Magazine, 22:57-62, 09 2001.
[27] S. Ross and D. Bagnell. Efficient reductions for imitation learning. In Proceedings of the thir- teenth international conference on artificial intelligence and statistics, pages 661-668. JMLR Workshop and Conference Proceedings, 2010.
[28] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In G. Gordon, D. Dunson, and M. Dudik, editors, Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 627-635, Fort Lauderdale, FL, USA, 11-13 Apr 2011 1. JMLR Workshop and Conference Proceedings.
[29] A. D. Dragan, K. Muelling, J. A. Bagnell, and S. S. Srinivasa. Movement primitives via optimization. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 2339-2346. IEEE, 2015.
[30] S. Schaal. Dynamic movement primitives-a framework for motor control in humans and hu- manoid robotics. In Adaptive motion of animals and machines, pages 261-280. Springer, 2006.
[31] Y. Tassa, S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, and N. Heess. dm_control: Software and tasks for continuous control, 2020.
[32] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu. Stable baselines. https: / httgg//hub..o//illla///aaaeeeeee
2018.
[33] O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarhhcccctttee reinforcement learning. CORR, abs/1 005.8005088808 082968 2018. URL http:::://arrii..rrg//a///1/.
[34] Haarnoja, Zhou, P. Abbeel, and S. Levine. Soft actor-eritic: Off-policy maximum entropy deep reinforeement learning with stochastic actor. CoRR, abs/1801.01290, 2018.
10

---

[35] L. Fan, Y. Zhu, J. Zhu, Z. Liu, O. Zeng, A. Gupta, J. Creus-Costa, S. Savarese, and L. Fei-Fei. Surreal: Open- reinforcement learning framework and robot manipulation benchmark. In Conference on Robot Learning, 2018.
[36] A. Mandlekar, F. Ramos, B. Boots, L. Fei-Fei, A. Garg, and D. Fox. Iris: Implicit reinforce- ment without interaction at scale for learning control from offline robot manipulation data. 2020 IEEE International Conference Robotics and Automation (ICRA), pages 44244420.
[37] A. Mandlekar, Y. Zhu, A. Garg, Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei. Roboturk: A crowdsoureing platform for robotic skill learning through imitation. In Conference Robot Learning, 2018.
[38] M. McDonald. Accelerate then imitate:
Learning from task and motion planning. Master's thesis, EECS Department, University of California, Berkeley, May 2021. URL http://wwu2. eecs. pocs.berin bew.eley.ey.................ele. http//uuusTTeehhpeee 2021//2222222222 EEEEEEE2S1-S90000 html.
[39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, N. Gomez, Kaiser, and I. Polo- sukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008, 2017.


Reviews:
[{"review_id": "WZ4kUpkaRf", "reviewer": "Reviewer_KfVB", "summary": "The authors propose a distributed training scheme for task and motion planning problems. This relies on access to symbolic robotics oriented task planner and motion planner to generate oracle training data for feed-forward policies that output task plans as action schemas and generate motion plans by chaining action schemas through reactive low-level control policies. The authors demonstrate the following:\n\n1. Ammortizing planning by approximating TAMP solvers through hierarchical policy networks results in a high success rate for TAMP tasks.\n\n2. Active exploration of problems space where the policy fails, and then generating oracle feedback can produce better quality data for optimization, by only using TAMP planner resources where feed-forward policies fail\n\n3. Feedforward policies can speed up a planning based approach by generating better initial guesses.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The idea of ammortizing planning through approximation is an elegant way of retaining the flexibility of iterative planning, and online repair by narrowing the exploration space using good quality feed-forward policies. The authors idea of a distributed framework appears to be a powerful method to achieve this.\n\n- The experiments are quite convincing towards the power of TAMP supervised feed-forward policies in comparison with RL-based or LfD-based policies. While this is to be expected as TAMP can provide a better training distribution, this work provides a convincing demonstration and a concrete modular implementation of this notion\n\n**Methodological comments**\n\n- The authors have restricted the action schemas to discrete outputs; the impact on problem domains that *require* continuous parameterizations in unclear. In theory,  the action selection network can output both discrete and continuous parameterizations, but the case of continuous parameters might be more challenging to train\n\n- There are a lot of hand-engineered inputs provided to the problem (constraints, fluent descriptions etc.) that are not available to an RL-based baseline. While this is a representational advantage of using structured methods, this does place the burden of generating these on an automation engineer. \n\n- The authors' decision to create a separate policy network for each action schema can potentially cause issues with more number of objects, shared parameters to reduce the memory requirements for control policy architectures would be interesting to look at.\n\n- The authors do not demonstrate how the policy networks performance is impacted by length of planning horizon required to solve the test problems. Does performance deteriorate when longer policies and task plans are required?\n\n**Presentation comments**\n\n- What problems are picked by the exploration nodes? Are they picked from a procedural problem distribution, a dataset of problems? The main paper does not address the role of the exploration nodes at all\n\n- Task planners are limited by the size of the logical states, the authors should include the number of logical fluents for each problem domain. Further, the authors do not provide any elaboration of how the geometrical facts for non-feasible task plans are generated. \n\n- I would appreciate a longer discussion of how this work relates to prior approaches in learning to approximate TAMP solvers. I found this discussion quite superficial and focusing on the implementation details rather than the underlying aspects of TAMP solvers that are being learned by the models. In particular the approach by Driess et al. is closely related, and provides similar capabilities, but the authors' distributed approach can potentially provide faster scaling. The reader would appreciate a deeper comparison with prior work. The authors can potentially tighten the text in the results section or move some of it to the supplementary materials to provide a better framing of their approach. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe that the idea of decentralized learning for approximating a TAMP solver is novel, yet the idea of approximating the TAMP solver by itself is not. I believe that the authors should frame their contribution as such\n\nFurther, the larger ideas are quite clear in the paper, but there are a lot of implementation details that are not elaborated enough.\n\nI don't believe the work is purely incremental.\n\nTherefore overall I recommend a weak accept, and I urge the authors to make some changes", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- Characterize the effect of planning horizon on the success rate. I would like to see how success rate deteriorates with longer required planning horizons\n\n- A deeper discussion on differences between prior work on approximating TAMP solvers and how your approach builds on or is novel in that context.\n"}}, {"review_id": "RwLq8xIucsZ", "reviewer": "Reviewer_1SbR", "summary": "This paper presents a framework for learning hierarchical long-horizon policies using a task and motion planner in the loop. The framework is based on a flexible TAMP algorithm, which uses an explicit interface between task planning and motion planning. Using this decoupled approach allows performing each part independently and in parallel. The task planner uses a Fast Forward algorithm to find sequences of actions, while motion planning is performed using numerical optimization.\n\nSince the output of the framework is a trained neural network policy, the TAMP plans are not necessarily carried out completely, rather they are used to generate datasets to train two policies (i.e., task and motion). These are in turn used to guide the search. \n\nHence the authors propose feedback mechanisms between the hierarchical policies and the TAMP algorithm which provide the training dataset for the policies. These feedback loops allow lowering the sample complexity of the framework. The proposed approach is thoroughly tested using three different benchmarks using ablation as way to highlight the important part of the framework.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and presents the ideas and results in clear way. The main strength of the paper is twofold.\n\n1)\tThe originality of the idea is very relevant and design choices are sound\n2)\tThe choice experiments are interested and show the potential of the framework\n\nEfficient motion policies for long-horizon manipulation tasks is clearly a very important milestone for robotics research. In this context very, few works combine learning and planning in an original way. As the authors point out most of the attempts use learning in the planning algorithms. In this work, the authors intertwine the two in such way that it minimizes sample complexity. For that using [15] seems very appropriate as it allows to decouple the task and motion planning problem.\n\nHaving experiments on three benchmarks allows to see, which parts of the framework are more important under which condition. For instance, considering feedback loops, the motion-feedback seems most important but sometimes task feedback can improve sample complexity in domains where the planner does not encounter the situations produced by the policy.\n\nWhile these are two strong points of the paper. They could be further improved by considering the following: The paper mainly lacks a theoretical situation of the problem. \n\nWhile it seems difficult to analyze theoretically this work, the framing of the research is not completely accurate. The title suggests that the authors are solely imitating the TAMP policies and that they use feedback to increase sample complexity. While that may be true in practice, it seems that a more accurate framing of the work is terms of Reinforcement Learning. The work can be viewed as a hierarchical version of Guided Policy Search (GPS). This way it should be possible to provide a better theoretical derivation and experimental analysis of the work.\n\nFor instance, taking inspiration from GPS, one could study the trade-off for generating policies that suddenly diverge from what the TAMP algorithm produces. This could be enforced in the motion optimization component using a cost function as done in GPS rather than only from rollout initialization. A dynamics model could be learned for the motion optimization, a KL constraint could be added in the supervised training phase etc. \n\nConcerning the theory, deriving the approach as some kind of descent algorithm would provide more clarity even if the implementation approximates it. \n\nFinally, concerning the experiments framing the problem as a model-based hierarchical RL algorithm naturally yields comparison to other hierarchical RL approaches such as HIRO, FuN etc. which are absent from the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It is recommended to frame this work in the context of Model-based hierarchical Reinforcement Learning, instead of active imitation learning. This reframing implies comparing to SOTA HRL algorithms such as HIRO or FuN. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See above.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "QF7T-Uw4tMo", "reviewer": "Reviewer_i7Gj", "summary": "The authors propose an imitation learning method for long-horizon robot tasks that is train on task and motion planning (TAMP) data. They train a hierarchical policy, with a high-level policy that learns to predict abstract/symbolic actions and a low-level policy which has an attention component and generates low-level motions. The goal is to generate a sequence of actions that both (1) generates high-level symbolic plans, and (2) generates trajectories.\n\nTraining is done via a system inspired by Dagger where they run a task and motion planner in parallel alongside training. After training, they run plans and if they fail, collect additional training examples by invoking the expert TAMP system.\n\nTraining runs as a set of nodes: the \"task planning\" node runs something like FastForward to generate a sequence of high-level actions, a motion planning module which generates trajectories, and training. A motion planning node generates trajectories, a training node updates the model, and finally they perform structured exploration to collect more data (similar to Dagger) to do both. This is the big innovation the authors propose, which allows them to efficiently train this hierarchical policy on a variety of simulated tasks.\n\nThey perform experiments on 3 simulated domains, 2 of which are from RGB images.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Briefly:\n+ Important problem\n+ Reasonable performance on common benchmarks\n+ Approach for training seems reasonable, and the distributed training architecture seems novel and helpful for large tasks where simulation is an option\n- numerous obvious typos \n- state space often not clear to me \n+ Good comparisons to related work\n\nI think the chief novelty is their distributed training architecture. The authors propose a system which shows performance on a range of different environments. They also show results both on RGB images and on joint state vectors and on a variety of problems and environments.\n\nI have some concerns about how this system would scale, though. The authors performed some ablation experiments were performed; however, these weren't performed in the cases where the system was trained on RGB images.\n\nThe meaning of the object ablation experiments in RoboSuite seemed unclear to me; the authors should explain this in more detail and what it says. They do an analysis of different types of feedback, which is interesting. It seems as if tamp-feedback is not useful more than motion-feedback, which might imply the high-level task structure is too simple.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this paper tries to solve an important problem and brings some interesting ideas, and they get decent results, even if the results are not demonstrated in the real world and the generalization experiments they perform are overly limited. The ablation experiments focus on feedback types, not on holding out different goals.\n\nThe distributed training system seems like it could be very helpful, though, if it could be applied to training models that could generalize or adapt to real problems.\n\nUPDATE: after the author's response I think it's more practical and stronger results than I had initially thought. Updated.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "\nMy big concern with this method is that it seems extremely data intensive and not very practical. The system as a whole seems to work on environments and tasks that have been seen; one of the big advantages of TAMP systems though is generalization to new problems. \n\nHow much data did your system end up needing? How many trajectories were collected? Some additional information would help determine how well it scales.\n\nAblation experiments holding out specific types of goals in RoboSuite would be very interesting and helpful.\n\nPlease make plots like Fig. 5 in color as well as using symbols; different shades of gray are very hard to parse.\n\nMinor:\nAbstract: let's --> lets\n3.1:  A symbolic planner, such as Fast\u0002123 Fowardciteff, --> FastForward~\\cite{...}\n"}}]

Author rebuttal:

Meta-review: This paper addresses the issue of jointly optimizing task and motion planning (TAMP), and proposed a method which includes to imitate a TAMP solver with a trained policy and to fast update the TAMP solver with a hierarchical policy architecture. 

I agree with reviewers that this paper is well organized and the idea is quite novel, the experimental results illustrated this sutdy is effective and promising. I believe this is a good work.

I also agree with reviewers on some minor points they mentioned, and suggest authors carefully address these comments to further improve this paper.

More added:
I would like to thank authors for your carefully and detailed responses to comments raised by reviewers. Authors responses seems to be effective, since the score of this paper was improved after rebuttle.
Also thanks feedbacks from reviewers.
I really believe the revised version is more better. 
Congratulations!

Decision: Accept (Oral)