Human-in-the-Loop Task and Motion Planning for Imitation Learning
Authors: Ajay Mandlekar
Caelan Reed Garrett
Danfei Xu
Dieter Fox
Venue: CoRL 2023 Poster (2023)
Keywords: Imitation Learning
Task and Motion Planning
Teleoperation

Abstract: Imitation learning from human demonstrations can teach robots complex manipulation skills, but is time-consuming and labor intensive. In contrast, Task and Motion Planning (TAMP) systems are automated and excel at solving long-horizon tasks, but they are difficult to apply to contact-rich tasks. In this paper, we present Human-in-the-Loop Task and Motion Planning (HITL-TAMP), a novel system that leverages the benefits of both approaches. The system employs a TAMP-gated control mechanism, which selectively gives and takes control to and from a human teleoperator. This enables the human teleoperator to manage a fleet of robots, maximizing data collection efficiency. The collected human data is then combined with an imitation learning framework to train a TAMP-gated policy, leading to superior performance compared to training on full task demonstrations. We compared HITL-TAMP to a conventional teleoperation system --- users gathered more than 3x the number of demos given the same time budget. Furthermore, proficient agents (75\%+ success) could be trained from just 10 minutes of non-expert teleoperation data. Finally, we collected 2.1K demos with HITL-TAMP across 12 contact-rich, long-horizon tasks and show that the system often produces near-perfect agents. Videos and additional results at https://hitltamp.github.io .

Human-In-The-Loop Task and Motion Planning for Imitation Learning
Ajay Mandlekar CCaelan Garrett", Danfei Xu Dieter Fox!
equal contribution, !NVIDIA, 2Georgia Institute of Technology
Imitation learning from human demonstrations can teach robots com- plex manipulation skills, but is time-consuning and labor intensive. contrast, Task and Motion Planning (TAMP) systems are automated and excel at solving long-horizon tasks, but they are difficult to apply to contact-rich tasks. In this pa- per, we present -omman-i--the-Love Task and Motion Plamning (HITL-TAMP), a novel system that leverages the benefits of both approaches. The system employs TAMP-gated control mechanism, which selectively gives and takes control to and from a human teleoperator. This enables the human teleoperator to man- age a fleet robots, maximizing data collection efficiency. The collected human data is then combined with an imitation learning framework to train a TAMP- gated policy, leading to superior performance compared to training on full task demonstrations. We compared HITL-TAMP to a conventional teleoperation sys- tem users gathered more than 3x the number of demos given the same time budget. Furthermore, proficient agents (75%+ success) could be trained from just 10 minutes of non-expert teleoperation data. Finally, we collected 2. K de- mos with HITL-TAMP across 12 contact-rich, long-horizon tasks and show that the system often produces near-perfect agents. Videos and additional results at https://hit Itam.. gi ii...o.
Abstract:
Keywords: Imitation Learning, Task and Motion Planning, Teleoperation
Introduction
Learning from human demonstrations has emerged as promising way to teach robots complex ma- nipulation skills [1, 2]. However, scaling up this paradigm to real-world long -orizon tasks has been difficult - providing long manipulation demonstrations is time-consuming and labor intensive [3]. At the same time, not all parts of a task are equally challenging. For example, significant portions of complex manipulation tasks such as part assembly or making a cup of coffee are free-space mo- tion and object transportation, which can be readily automated by non-learning approaches such motion planning. However, planning methods generally require accurate dynamics models [4] and precise perception, which are often unavailable, limiting their effectiveness contact-rich and low-tolerance manipulation. In this context, our work aims at solving real-world long-horizon ma- nipulation tasks by combining the benefits of learning and planning approaches.
Our method focuses on augmenting Task and Motion Planning (TAMP) systems, which have been shown to be remarkable at solving long-horizon problems [5]. TAMP methods can plan behavior for wide range of multi-step manipulation tasks by searching over valid combinations of small number of primitive skills. Traditionally, each skill is hand-engineered; however, certain skills, such as closing a spring-loaded lid or inserting a rod into a hole, are prohibitively difficult to model in productive manner. Instead, we use combination of human teleoperation and closed-loop learning to implement just these select skills, keeping the rest automated. These skills use human teleopera- tion at data collection time and a policy trained from the data at deployment time. Integrating TAMP
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

systems and human teleoperation poses key technical challenges special care must be taken to enable seamless handoff between them to ensure efficient use of human time.
To address these challenges, we introduce Human-in-the-Loop Task and Motion Plan- ning (HITL-TAMP), system that symbioti- cally combines TAMP with teleoperation. The system collects demonstrations by employing a TAMP-gated control mechanism - it trades off control between a TAMP system and a hu- man teleoperator, who takes over fill in gaps that TAMP delegates. Critically, human opera- tors only need engage at selected steps of task plan when prompted by the TAMP system, meaning that they can manage fleet of robots by asynchronously engaging with one demon- stration session at a time while TAMP system controls the rest of the fleet.
In In-pood, sachrintt Close 2
Tosk Goaal
lask and Motion Planning
goal
start
Tesk Flan
F2
Motor Biills
Rg
Motion plan/ motar primitives
Fo iikes trained from human data
Figure 1: Overview. HITL-TAMP decomposes task (making coffee) into planning-based (TAMP) and learning-baset (human imitation) segments.
By soliciting human demonstrations only when needed, and allowing for human participate in multiple parallel sessions, system greatly increases the throughput of data collection while lowering the effort needed to collect large datasets on norghooiioon contact-rich tasks. We combine our data collection system with an imitation learning framework that trains a TAMP-gated policy (as illustrated in Fig. 1) on the collected human data. We show that this leads to superior performance compared to collecting human demonstrations on the entire task, terms of the amount data and time needed for a human teach a task the robot, and the success rate of learned policies.
The main contributions of this paper are:
We develop HITL-TAMP, an efficient data collection system for long horizon manipulation tasks that synergistically combines and trades off control between a TAMP system and a human operator.
HITL-TAMP contains novel components including (1) a mechanism that allows TAMP to learn planning conditions from small number of demonstrations and (2) queuing system that allows demonstrator manage a fleet of parallel data collection sessions.
We conduct study (15 users) compare HITL-TAMP with conventional teleoperation system. Users collected over 3x more demos with our system given the same time budget. Proficient agents (over 75% success) could be trained from just 10 minutes of non-expert teleoperation data.
We collected 2. IK demos with HITL-TMMP across 12 contact-rich and long-horizon tasks, includ- ing real-world coffee preparation, and show that HITL-TAMP often produces near-perfie sear-perfect agents.
2 Preliminaries
Summary of Related Work. Several works have shown the value in learning robot manipulation
with human demonstrations [6, 1, 2, 7, 9, 10, 11, 8), in developing automatic control hand-offs between human supervisor and automated system for more effective data collection [12, 13, 14, 15, 16], and in combining learned and predefined skills [17, 18, 19, 20. 21]. Prior TAMP [5, 22, 4, 23] works have also integrated sarning-based components [24, 25, 26, 27, 28, 29, 30, 31, 32] to make less assumptions on prior knowledge. See Appendix D for full related work.
Problem Statement. We consider robot acting in discrete-time Markov Decision Process (MDP) (X, T(x' x, u), R(x), Po) defined by state space X, action space U, transition dis- tribution T, reward function R, and initial state distribution Po. We assume we are given an offline dataset of N partial demonstration trajectories (collected via our HITL-TAMP system, see Sec. 3.3)
D = {(r u6). (xf.u() Ti JN We train policies я with Behavioral Cloning [33] using the objective arg ming (Zz,))) | (x) ) ull2 (details in Appendix K).
We consider a TAMP policy (u æ) for controlling the robot. It plans sequence of actions that will be tracked using feedback controller. We use the PDDLStream [23] planning framework,

---

logic-based action language that supports planning with continuous values, to model our TAMP do- main. States and actions are described using predicates, Boolean functions, which can have discrete and continuous parameters. A predicate paired with values for its parameters is called a literal. Our TAMP domain uses the following parameters: o is an object, 9 E SE(3) a 6-DoF object grasp pose relative to the gripper, p E SE(3) is an object placement pose, q é Rd a robot configuration with d DoFs, and is robot trajectory comprised of sequence of robot configurations.
The planning state is a set of true literals for fluent predicates, predicates who's 's truth value can change over time. We define the following fluent predicates: AtFose((, p) is true when object o is placed at placement p; At ArGrasp g) is true when object o is grasped using grasp gi At Con (q) true when the robot is at configuration qi Empty() true when the robot's 's end effector empty; Attachedi(o, o') is true when object is attached to object o':
We use the Tool Hang task as running example (see Fig. 5), where the robot must insert frame into a stand and then hang a tool on the frame. The set of goal system states X, is ex- pressed as a logical formula over literals. Let so be the initial state so and G be the goal formula:
So (AtPose(frame, pO) At AtPose(tool, po), At At.ooee(saani po), AtConf(qo), Empty())
G = At Attached[frmme, stand) A Attached(tool, frame) A Empty()
Planning actions a are represented using action schemata. An action schema is defined by a 1) name, 2) list of parameters, 3) list of static mion-fluent) literal constraints (con) that valid parameter values satisfy, 4) list of fluent literal preconditions (pre) that must hold to correctly execute the action, and 4) list of fluent literal effects (eff) that specify changes to state. The move action advances the robot from configuration 91 to Q2 via trajectory т. The constraint Mot onttion , т, 92) is satisfied if qı and d2 are the start and end of т. the pick action, the constraint Grasp(o, g) holds if is a valid grasp for object o, and the constraint Pose(o, p) holds p a valid placement for object o. The explicit constraint [g) g = p represents kinematies, namely that forward kinematics f Rd SE((3 for the robot's 's gripper from configuration g multiplied with grasp g produces pose p.
move( 72
pick(o, g.p,
con: [Motion(yn,T.ee T242
con: |Grasp(o, g), Pose(o,p),
pre: (AtConf(q)), Safe(r)]
pre: [AtPose(o, p), Empty(, AtConf(g)
eff: (AtConf(q:), -AtConf(g))
eff: (AtGrasp(o,g), -AtPose(o,p). -Empty(]
The limitations of the TAMP system are that, although it can readily observe the robot state, it does not have the ability to precisely estimate the environment and productively react to changes in it in real-time. Thus, it's advantageous to teleoperate skills that require 1) contact-rich interaction that is difficult to accurately model and 2) precision greater than that which the perception system can deliver. An example of 1) is the insertion phase of Tool Hang, which typically requires contacting the walls of the hole align the frame, and an example of is the hanging phase of Tool Hang. which requires precisely aligning the hole of the tool with the resting frame.
3 Integrating Human Teleoperation and TAMP
To make TAMP and conventional human teleoperation systems compatible, describe crucial components that allow for seamless handoff between TAMP and human operator. These include 1) novel constraint learning mechanism that allows TAMP to plan to states that enable subsequent human teleoperation (Sec. 3.2) and 2) the core TAMP-gater teleoperation algorithm (Sec. 3.3).
3.1 Teleoperation Action Modeling
To account for human teleoperation during planning, we need an approximate model of the teleop- eration process. We build on the high-level modeling approach of Wang et al. [25] by specifying an action schema for each skill identifying which constraints can be modeled using classical techniques. Then, extract the remaining constraints from handful of teleoperation trajectories. Continuing running example, we teleoperate the frame insertion and tool hang in the Tool Hang task.
3

---

The attach action models any skill that involves attaching one movable object to another object, for example, by placing, inserting, or hanging. Its parameters are a held object o, the current grasp g for o, the corresponding current pose of o, the current robot configuration pose of o, the subsequent robot configuration 7. and the object to be attached o'. This action stochastic as the human teleoperator "chooses" the resulting pose and configuration q (indicated d'). Rather than explicitly model
the subsequent
by [), which modeled by the constraint HumanAttach this constraint, we take an optimistic determinization of the outcome by assuming that the human produces satisficing P. 7 pair, without committing to specific numeric values.
attach(o, 9, P. 9.4.9.
con: [AttachGrasp(o..]] Preattach(o.p."), (f(g) * pl,
Goodkttach(o,P.o)) HumanAttach(o.....00 pre: (AtGrasp(o.g), AtConf(g)
eff: (AtPose(u,9), Empty(, Attached(o,o), AtConf(ā), -AtGrasp(o.g) g),, (AtConf(4))
The key constraint is GoodAttach(on, Pr o'), which is true if object o at pose satisfies the ground- truth goal attachment condition in 9 with object o'. The human teleoperator is tasked with reaching a pose p that satisfies this constraint, which postcondition the action. The goal of model learning is to represent the preconditions (Sec. 3.2) that facilitate this in a generative fashion.
3.2 Constraint Learning
To complete the action model, we learn the AttachGrasp and PreAttach constraints, which involve parameters in attach's precon- ditions. We bootstrap these constraint models from few (~ 3 in our setting) human demon- strations. These demonstrations only need to showcase the involved action. Through com- positionality, these actions can be deployed in many new tasks without the need for retraining. In this work, because the set of objects fixed, the constraints do not need to generalize across ob- jects so we simply use uniform distributions over pose datasets conditioned on task and objects. In settings where there are novel objects at test time, we could instead estimate these affordances across objects directly from observations [25, 34, 35] using more complicated (deep) generative models.
Partal Human Damenatnatire
PENKEcach-Paan Phamo grand) Conetmint Dietibutions
PreAttaciioo framol
Figure 2: Constraint learning. Example of learned at- tach conditions for the frame (leff) and tool from hand- ful of demonstrations for the Tool Hang task.
We define P PreAttach(o, p. d') to be true if is pose for object immediately prior to the human achieving GoodAttach( P, o'). For each human demonstration, we start at the first state where GoodAttach is satisfied and then search backward time for the first state where (I) the robot holding object and (2) objects o and are least 8 centimeters apart. This minimum distance constraint ensures that and o' are not in contact in manner that is spatially consistent and robust to perception and control error. We log the relative pose p between and o' data point and continue iterating over human demonstrations to populate dataset Pg {p PreAttach( p, o')}.
Similarly, we define AttachGrasp(( g) to be true if g is a grasp for object o allows for the human achieving foodAttachi
o'). Not all object grasps enable the human to satisfy the target condi- tion, for example, a frame grasp on the tip that needs be inserted. Similar to PreAt tach, for each demonstration we log the relative pose between the robot end effector and object the first pre- contact state before satisfying GoodAttach producing dataset G {g AttachGrasp(o, g)}.
3.3 TAMP--Gated Teleoperation
We now describe TAMP-gated telcoperation, where a TAMP system decides when to execute por- tions of a task, and when human operator should complete portion (full details Appendix J). Each teleoperation episode consists of one or more handoff's where the TAMP system prompts human operator to control portion of task, or where the TAMP system takes control back after it determines that the human has completed their segment.

---

Human-in-the-lop TAMP Data Collection Sessions
R1
R2
R3
RM
F15
RB
DI
TAMP
TAMP
TAMP
Dequeus + Contri Switch
FIFO Queus
Teleoperation
Figure 3: Queueing system. HITL-TAMM's queueing system allows human teleoperator (bottom left) manage flect of syynchronously-rumin data collection sessions (RI-66).
Every task is defined by a goal formula G. On each TAMP iteration, it observes the current state 8. it satisfies G the episode terminates, otherwise, the TAMP system solves for a plan ā from current state s to the goal G. TAMP subsequently issues joint position commands to carry out planned mo- tions until reaching an action a requiring the human. Next, control switches into teleoperation mode, where the human has full 6-DoF control of the end effector. We use smartphone interface similar to prior teleoperation systems [36, 37, 10]. The robot end effector is controlled using an Operational Space Controller [38]. The TAMP system monitors whether the state satisfies the planned action postconditions a. effects. Once satisfied, control switches back to the TAMP system, which replans.
Scaling Data Collection for Learning
Increasing Data Throughput with Queueing System. Since the TAMP system only requires human assistance small parts of an episode, a human operator has the opportunity to manage multiple robots and data collection sessions simultaneously. To this end, we propose novel queue- ing system (Fig. 3) allowing each operator interact with fleet of robots. We implement this by using several (Nutta) robot processes, single human process, and queue (more analysis in Appendix 1). Each robot process runs asynchronously, and spends its time in I of 3 modes - (1) being controlled by the TAMP system, (2) waiting for human control, or (3) being controlled by the human. This allows the TAMP system to operate multiple robots parallel. When the TAMP sys- tem wants to prompt the human for control, it enqueues the environment into the shared queue. The human process communicates with human teleoperation device and sends control commands to one robot process time. When the human completes segment, TAMP resumes control of the robot, and the human process dequeues the next session from the queue.
TAMe- Policy Deployment. HITL-TAMP results in demonstrations that consist of TAMP- controlled parts and human-controlled parts - we train a policy with Behavioral Cloning [33] on human portions (details Appendix K). To deploy the learned agent, we use a TAMP-gated control loop that identical the handoff logic Sec. 3.3, using the policy instead of the human.
Experiment Setup
Tasks. We chose evaluation tasks that are contact-rich and long-horizon, validate that HITL- TAMP indeed combines the benefits of the two paradigms (see Fig. 4 and Fig. 5). We further evaluated HITL-TAMP variants of tasks where objects are initialized in broad regions of the workspace, difficult setting for imitation learning systems in the past. Full details in Appendix E.
Pilot User Study. We conducted a pilot user study with 15 participants to compare our system (HITL-TAMP) to a conventional teleoperation system [36], where task demonstrations were col- lected without TAMP involvement. Each participant performed task demonstrations on 3 tasks (Coffee, Square (Broad), and Three Piece Assembly (Broad)) for 10 minutes each system,
5

---

(a) Square
(b) Coffee
(c) 3 Pc. Assembly
(d) Tool Hang
(e) Coffee Prep.
Figure 4: Tasks. We use HITL-TAMP to collect demonstrations for contact-rich, Long-horizon tasks.
Demos (avg-user)
(ang novice)
(avg-user)
(avg-minvice
Coffee (C)
11.2
7.2
168.0)
24.4
15.0
76.0
Coffee (HT)
131.0
100.0
Square Broad (C)
11.1
5.2
186.0
0.0
20.0
Square Broad (HTT)
49.5
41.8
747.0
80.0
77.5
98.0
Three Piece Assembly Broad
7.0
117.0
Three Piece Assembly Broad (HT)
15.1
8.0
227.0
27.7
17.5
66.0
Table User Study Data Collection and Policy Learning Results. We report number of demos collected averaged across users (avg-user), averaged across novice users (avg-novice), and summed across all users (all). We also report the success rate of policies trained per-user data (avg-user: averaged across all users, and avg- novice: averaged across novice users), and trained all user data (all). Users collected more demonstrations using HITL-TAMP (HT)
conventional system (C), and policy performance was vastly greater well.
totaling 60 minutes of data collection across the tasks and systems. Participants filled out post-study survey rank their experience with both systems. Each participant's number of success- ful demonstrations was recorded to evaluate the data throughput of each system, and agents were trained on each participant's 's demonstrations and across all participants' demonstrations (Sec. 6.1).
Experiment Results
We (1) present user study results to highlight HITL-TAMP's data collection efficiency (Sec. 6. 6.1), (2) compare trained HITL-TAMP agents policies trained from full task demonstrations (Sec. 6.2), and (3) deploy HITL-TAMM in the real world without precise perception (Sec. 6.3).
6.1 System Evaluation: User Study
We show that (1) HITL-TAMP allows participants to collect demonstrations much faster than conventional teleoperation, (2) with varying system proficiency, (3) HITL-TAMP enables novice operators to collect high-quality demonstration data, and (4) HITL-TAME requires less user effort than conventional teleoperation.
train performant policies using data collected from users
HITL-TAMP enables users to collect task demonstrations at a much higher rate than a con- ventional teleoperation system. As Table shows, collectively, our 15 users gathered 2.5x more demonstrations with HITL-TAMP when compared the conventional system the Coffee task (431 vs. 168), 4.5x more Square Broad (747 166), and nearly 2x more Three Piece As- sembly Broad (227 vs. 117). The high collection efficacy of HITL-TAMP was also reflected per-user basis users averaged 28.7 demos on Coffee (vs. 11.2), 49.8 demos on Square Broad (vs. 11.1), and 15.1 demos on Thrre Piece Assembly Broad (vs. 7.8), during their 10-minute sessions.
HITL-TAMP enables performant policies to trained from minutes of data. We used each per- son's 10-minute demonstrations train policy for each (user-task) pair with behavioral cloning. Agents trained on HITL-TAMP data vastly outperformed those trained from the conventional tele- operation data (Table 1) - agents achieved an average success rate of 90.7% on Coffee (vs. 24.4%), 80.0% on Square Broad (vs. 1.2%), and 27.7% on Three Piece Assembly Broad (vs. 0.0%).
HITL-TAMP enables training proficient agents from multi-user data. Prior work [39, 1] noted that imitation learning from multi-user demonstrations can be difficult. However, we found agents trained on the full set of multi-user HITL-TAMI data achieve high success rates (100.0%. 98.0%, and 66.0% on Coffee, Square Broad, and Three Piece Assembly Broad, respectively) compared to

---

Task
Success
TAMP
Stack Three
62.0
Coffee
74.0
Coffee Broad
68.0
Tlrran
Human
Tool Hang
64.0
TAMP
Hurton
TRMP
Humen
Figure S: (left) Real Tasks. Coffee (top), version where the machine be either side (Coffee Broad, not shown). Stack Three (middle), and Tool Hang (bottom). show TAMP orange and human in blue. (right) Real World Policy Performance. We collected 100 demonstrations on Stack Three, Coffee, and Coffee Broad and 50 demonstrations on Tool Hang with HITL-TAMP and report the policy performance in this tablc.
Time (min)
(lw-ddim))
(image)
Task
Time (min)
SR (Im)
TAMP-gated SR (im)
Square
Square
14.0
Square
[1]
100.0
Culfie
Coffee Broadd
25.8
22.6
10300
08.3 0.0
100.0:00.3
16.7 0.9
Broad
94.7
Square Broad (IT)
14.0
100.0 1.0.
100.0 0.0
Tad Hung
68.0
9.
7..7+-0.5
Tod Hung Broal
51.5
43.3 1.9
40.7 0.0.
Three Piece Assembly (C)
60.0
73.3 1
17.3 1.7
Three Place Axsemibyy
91.0
1030 0.0
100.0 0.0
Three Pleve Assembly (ITT)
30.0
100.0 1: 0.0
100.0 0.0
Those Plece Avecublyy Rroad
$1.9
84.7 1.1
52.0 1.6
Toal Mang (C) [1]
80.0
67.3 0.0
82.0 2.8
Coffee Preperation
78.4
96.0
100,0 d 0.0
Tool Hang (HI)
48.0
78.7 + 0.9
78.7 0.9
Figure 6: (left) Results HITL-TAMP datasets. We collected 200 demonstrations each task with HITL- TAMP and trained low-dim and visuomotor agents TAMP-gated agents on each dataset. (right) Comparison to conventional teleoperation datasets. We trained both normal and TAMP-gare gated policies using conventional teleoperation (C) and compared them to HITL-TAMP (HT). Surprisingly, TAMP TAMP-gating gating makes policies trained on the data comparable to HITL-TAME data, but data collection still involves significantly higher operator time. those trained on the full set of conventional teleoperation data (76.0%, 20.0%, 0.0%) (see Table 1). In fact, the worst per-user HITL-TAMP policy (10-minutes of data) outperformed the policy trained on the full set of conventional telcoperation data (150 minutes) on both Square Broad (56.0% vs. 20.0%) and Three Piece Assembly Broad (14.0% vs. 0.0%).
HITL-TAMP enables non-experts to demonstrate tasks efficiently. of the 15 users in our study had no experience with teleoperation. Table shows we found that they were able to collect far more data average with HITL-TAMP (more than 3x on Coffee, more than 8x Square Broad) and policies trained on their HITL-TAMP data achieved significantly higher success the conventional system 90.0% (vs. 15.0%) on Coffee, 77.5% (vs. 0.0%) on Square Broad, and 17.5% (vs. 0.0%) on Three Piece Assembly Broad.
HITL-TAMP results in a lower perceived workload compared to the conventional teleopera- tion system. Each participant completed a NASA-TLX survey [40] to rank their perceived workload for each system across 6-categories (100-point scale, increments of 5). Users found HITL-TAMP to require less mental demand (36% vs. 74%), less physical demand (29.7% vs. 63.7%), and less tem- poral demand (28.3% vs. 53.7% 7)),,, while enabling higher overall performance (83.7% vs. 59.7%), with lower effort (29.3% vs. 75.7%) and lower frustration (30.0% vs. 65.0%).
6.2 Learning Results
We collect datasets with HITL-TAMP across tasks (see Sec. 5) and show that highly capable policies can be trained from this data. The results compare favorably to training on equal amounts of demonstrations from a conventional teleoperation system.
HITL-TAMP is broadly applicable to wide range of contact-rich and long-horizon tasks. Using HITL-TAMP, we had a single human operator collect 200 demonstrations on each of our tasks. We then trained agents from this data on two observation spaces - low-dim observations, where agents directly observe the poses of relevant objects, and image observations, where agents observe a front- view RGB image and wrist RGB image (as in [1]). Table shows that across both observation spaces, HITL-TAME trains near-perfect agents on several tasks (Square, Coffee, Three Piece Assembly), including broad tasks with wide distribution of object initialization (Square Broad, Coffee Broad, Three Piece Assembly Broad). HITL-TAMP also achieves high performance on the Tool Hang task (80.7% low-dim, 78.7% image), which is the hardest task in the robomimic
7

---

benchmark [1]. It is also able to train performant agents (49.3% low-dim, 40.7% image) on broad version of the task (Tool Hang Broad). Finally, HITI--TAMP trains near-perfect agents (96%) on the Coffee Preparation task, which consists of several stages (4 TAMP segments and policy segments) involving low-tolerance mug placement, drawer grasping and opening, lid opening, and pod insertion and lid closing.
HITL-TAMP compares favorably to conventional teleoperation systems in terms of operator time and policy learning, Even when an equal number of task demonstrations are used, learned policies from HITL-TAMM still outperform those from conventional teleoperation. We run our com- parison on 4 tasks - Square, Square Broad, Three Piece Assembly, and Tool Hang. where each task has 200 HITL-TAMP demos collected and 200 conventional system demos. As Table 6 shows, HITL-TAMP enabled collecting 200 demonstrations each task in much shorter periods of time (additional analysis in Appendix F). Furthermore, agents trained on HITL-TAMP data outperform agents trained on conventional data (with the largest gap being 100.0% vs. 15.3% Square Broad).
TAMP-gated control a crucial component to train proficient policies. We took the 200 demon- stration datasets collected via conventional teleoperation, trained the agents as normal, but deployed them with TAMP-gated control during policy evaluation. This dramatically increases their success rates and gives comparable results HITL-TTAMP data (see Table 6). This shows that datasets con- sisting of entire human demonstration trajectories compatible with TAMP-gated control. How- ever, they remain time-consuming to collect, and HITL-TAMP greatly reduces the time needed.
6.3 Real Robot Validation
We apply HITL-TAMP to a physical robot setup with a robotic arm, a front-view camera, and a wrist-mounted camera. The only significant change from simulation the need for perception to obtain pose estimates the objects to populate the TAMP state. We do not assume any capability to track object poses in real-time. Instead, allow the human to demonstrate (and the policy to imitate) behaviors from partial observations (RGB cameras). We collected 100 demonstrations for each of 3 tasks - Stack Three, Coffee, and Coffee Broad, and 50 demonstrations on Tool Hang, and report policy learning results across 50 evaluations for each task (25 for Tool Hang) (see Fig. 5). Our HAMP-gated agent achieves 62% on Stack Three, 74% on Coffee, 66% on Coffee Broad (72% with the machine on the right side of the table, and 60% with the machine on the left side), and 64% on Tool Hang (as opposed to the 3% from 200 human demonstrations in prior work [1]).
7 Limitations
See Appendix for full limitations. We assume tasks can described in PDDLStream and that human teleoperators can demonstrate them. The tasks in this work focus on tabletop domains with limited object variety future work could scale HITL-TAMP to more diverse settings. Currently, HITL-TAMP requires prior information (at high-level) on which task portions will be difficult for TAMP. We also assume access to coarse object models and approximate pose estimation to conduct TAMP segments in the real world. Future work could relax these assumptions by integrating perception uncertainty estimates, and extending TAMP to not require object models [35].
Conclusion
We presented a new approach to teach robots complex manipulation skills through hybrid strategy of automated planning and human control. Our system, HITL-TAMP, collects human demonstra- tions using TAMP-gated control mechanism and learns preimage models of human skills. This allows for human to efficiently supervise team of worker robots asynchronously. The combi- nation of TAMP and teleoperation in HITL-TAMP results in improved data collection and policy learning efficiency compared to collecting human demonstrations on the entire task.

---

Acknowledgments
This work was made possible due the help and support of Sandeep Desai (robot hardware), Ravin- der Singh (IT), Alperen Degirmenci (compute cluster), Yashraj Narang (valuable discussions), An- ima Anandkumar (access to robot hardware), Yifeng Zhu (robot control framework [41]), and Shuo Cheng (drawer design used in Coffee Preparation task). We also thank all of the participants of our user study for contributing their valuable time and feedback on their experience.
References
[1] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
[2] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint urXiv:2212.066.
2022.
[3] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard, Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 3: 297-330, 2020.
[4] M. A. Toussaint, K. R. Allen, K. A. Smith, and J. B. Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. 2018.
[5] C. Garrett, R. Chitnis, Holladay, B. Kim, T. Silver, P. Kaelbling, and Lozano- Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-299 2021.
[6] Zhang, McCarthy, O. Jow, D. Lee, K. Goldberg, and Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. arXiv preprint 17Xiv:1710.04600 15.0415, 2017.
[7] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learn- ing, pages 991-1002. PMLR, 2022.
[8] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. Conference on robot learning, pages 11 1113-1132. PMLR, 2020.
[9] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time. arXiv preprint arXiv:2210.06400 2022.
[10] A. Mandlekar, D. Xu, R. Martín-Marth S. Savarese, and L. Fei-Fei. Learning general- ize across long-horizon tasks from human demonstrations. arXiv preprint arXiv:2003.0661 06085, 2020.
[11] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as can, not say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.00 06691, 2022.
[12] R. Hoque, A. Balakrishna, C. Putterman, M. Luo, D. S. Brown, D. Seita, B. Thananjeyan, E. Novoseller, and K. Goldberg. Lazydagger: Reducing context switching in interactive im- itation learning. In 2021 IEEE 17th International Conference Automation Science and Engineering (CASE), pages 502 509. IEEE, 2021.
[13] R. Hoque, L. Y. Chen, S. Sharma, K. Dharmarajan, Thananjeyan, P. Abbeel, and K. Gold- berg. Fleet-dagger: Interactive robot fleet learning with scalable human supervision. In Con- ference Robot Learning, pages 368-380 PMLR, 2023.

---

[14] J. Zhang and K. Cho. Query-efficient imitation learning for end-to-end autonomous driving. arXiv preprint 1605.06400 2016.
[15] R. Hoque, A. Balakrishna, Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. arXiv preprint S1t.::2009008273. 2021.
[16] S. Dass, K. Pertsch, H. Zhang, Y. Lee, J. J. Lim, and S. Nikolaidis. Pato: Policy assisted telcoperation for scalable robot data collection. arXiv preprint arXiv:2212.04703 2022.
[17] T. Silver, K. Allen, J. Tenenbaum, and L. Kaelbling. Residual policy learning. arXiv preprint arXiv:1812.06298, 2018.
[18] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and Levine. Residual reinforcement learning for robot control. In 2019 International Conference on Robotics and Automation (ICRA), pages 6023 6029. IEEE, 2019.
[19] A. Kurenkov, A. Mandlekar, R. Martin-Martin, S. Savarese, and A. Garg. Ac-teach: A bayesian actor-critic method for policy learning with an ensemble of suboptimal teachers. arXiv preprint arXiv: /21 2019.
[20] O. Mces, Borja-Diaz, and W. Burgard. Grounding language with visual affordances over unstructured data. arXiv preprint arXiv:2210.01911, 2022.
[21] E. Valassakis, N. Di Palo, and E. Johns. Coarse-to-fine for sim-to-real: Sub-millimetre pre- cision across wide task spaces. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5989-5996. IEEE, 2021.
[22] L. P. Kaclbling and T. Lozano-Péret Hierarchical task and motion planning in the now. In ICRA, 2011.
[23] C. R. Garrett, Lozano-Pérez and P. Kaelbling, Pddistream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pages 440-448 2020.
[24] G. Konidaris, L. P. Kaelbling, and T. Lozano-Perez. From skills to symbols: Learning symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61: 215-289, 2018.
[25] Z. Wang, C. R. Garrett, L. P. Kaelbling, and T. Lozano-Pérez. Learning compositional models of robot skills for task and motion planning. The International Journal of Robotics Research, 40(6-7):866-88)) 894, 2021.
[26] Liang, M. Sharma, A. LaGrassa, S. Vats, S. Saxena, and O. Kroemer. Search-based task planning with learned skill effect models for lifelong robotic manipulation. In 2022 Interna- lional Conference on Robotics and Automation (ICRA), pages 6351-6357. IEEE, 2022.
[27] H. M. Pasula, L. S. Zettlemoyer, and L. P. Kaelbling. Learning symbolic models of stochastic domains. Journal of Artificial Intelligence Research, 29:309-352, 2007.
[28] Silver, R. Chitnis, J. Tenenbaum, L. Kaelbling, and Lozano-Pérez. Learning sym- bolic operators for task and motion planning. In 2021 IEEE/RS International Conference Intelligent Robots and Systems (IROS), pages 3182-3189, IEEE, 2021.
[29] R. Chitnis, D. Hadfield- Menelll, A. Gupta, S. Srivastava, E. Groshev, C. Lin, and P. Abbeel. Guided search for task and motion plans using learned heuristics. In ICRA. IEEE, 2016.
[30] B. Kim, L. Shimanuki, L. P. Kaelbling, and T. Lozano-Pérez. Representation, learning, and planning algorithms for geometric task and motion planning. IJRR, 41(2), 2022.
10

---

[31] S. Cheng and D. Xu. Guided skill learning and abstraction for manipulation. arXiv preprint arXiv:2210. 12.2200..26.,,. 12631, 2022.
[32] T. Silver, A. Athalye, J. B. Tenenbaum, T. Lozano-Perez, and L. P. Kaelbling. Learning neuro- symbolic skills for bilevel planning. arXiv preprint arXiv:2206. 2022.
[33] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Advances in neural information processing systems, pages 305-313, 1989.
[34] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspner Efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA). pages
IEEE, 2021.
[35] A. Curtis, X. Fang. L. Kaelbling. T. Lozano-Pérez, and R. Garrett, Long-horizon manip- ulation of unknown objects via task and motion planning with estimated affordances. In 2022 International Conference on Robotics and Automation (ICRA), pages 1940-1946. IEEE, 2022.
[36] A. Mandlekar, Y. Zhu, A. Garg, Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei. RoboTurk: A Crowdsourcing Platform for Robotic Skill Leaming through Imitation. In Conference on Robot Learning. 2018.
[37] A. Mandlekar, J. Booher, M. Spero, A. Tung, A. Gupta, Y. Zhu, A. Garg, S. Savarese, and L. Fei-Fei. Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity. arXiv preprint 1rXiv:1911.04000 2019.
[38] O. Khatib. A unified approach for motion and force control of robot manipulators: The opera- tional space formulation. IEEE Journal on Robotics and Automation, 3(1):43-53, 1987.
[39] A. Mandlekar, F. Ramos, B. Boots, Savarese, L. Fei-Fei, A. Garg, and D. Fox. Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data. In IEEE International Conference on Roboties and Automation (ICRA), pages 4414- 4420. IEEE, 2020.
[40] G. Hart and E. Staveland. Development of nasa-tlx (task load index): Results of empirical and theoretical research. In Advances in psychology, volume 52, pages 139-183. Elsevier, 1988.
[41] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. 6th Annual Conference on Robot Learning, 2022.
[42] O. Mees, L. Hermann, E. Rosete-Beatt Beas, and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3)77327-7334 2022.
[43] A. Mandlekar, D. Xu, R. Martin-Martin, Zhu, L. Fei-Fei, and S. Savarese. Human-in-the loop imitation learning using remote telcoperation. arXiv preprint arXiv:2012.06733, 2020.
[44] K. Van Wyk, M. Culleton, J. Falco, and K. Kelly. Comparative peg-in-hole testing of a force- based manipulation controlled robotic hand. IEEE Transactions on Robotics, 34(2):542-55) 549, 2018.
[45] H. Park, J. Park, D.-H. Lee, J.-H. Park, and J.-H. Bae. Compliant peg-in-hole assembly using partial spiral force trajectory with tilted peg posture. IEEE Robotics and Automation Letters, 5(3):4447-444 4454, 2020.
[46] M. J. McDonald and D. Hadfield-Mened Guided imitation of task and motion planning. In Conference Robot Learning, pages 630-640. PMLR, 2022.

---

[47] M. Kelly, C. Sidrane, K.
and M. J. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pages 8077-8083. IEEE, 2019.
[48] J. Spencer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang, P. Ramadge, and S. Srinivasa. Learning from interventions: Human-robot interaction as both explicit and implicit feedback. In 16th Robotics: Science and Systems. RSS 2020. MIT Press Journals, 2020.
[49] Q. Li, Z. Peng, and B. Zhou. Efficient learning of safe driving policy via human-ai copilot optimization. In International Conference on Learning Representations, 2021.
[50] J. S. Warm, R. Parasuraman, and G. Matthews. Vigilance requires hard mental work and is stressful. Human factors, 50(3):433-444 441, 2008.
[51] K. Menda, K. proach to safe imitation learning. In 2019 /EEF/RSJ International Conference Intelligent Robots and Systems (IROS), pages 5041-5048. IEEE, 2019.
and M. J. Kochenderfer. Ensembledagger: A bayesian ap-
[52] T. Mandel, Y.-E. Liu, E. Brunskill, and Z. Popović. Where to add actions in human-in-the- loop reinforcement learning, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[53] A. Jonaavittula and D. P. Losey. Learning to share autonomy across repeated interaction. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1851-1858. IEEE, 2021.
[54] R. Tedrake, M. Fallon, S. Karumanchi, S. Kuindersma, M. Antone, T. Schneider, T. Howard, M. Walter, H. Dai, R. Deits, al. A summary of team mit's s approach the virtual robotics challenge. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 2087-2087. IEEE, 2014.
[55] R. Luo, C. Wang, E. Schwarm, C. Keil, E. Mendoza, P. Kaveti, S. Alt, H. Singh, T. Padir, and J. P. Whitney. Towards robot avatars: Systems and methods for teleinteraction at avatar xprize semi-finals. In 2022 IEEE/RS International Conference on Intelligent Robots and Systems (IROS), pages 7726-7733. IEEE, 2022.
[56] J. M. Marques, N. Patrick, Y. Zhu, N. Malhotra, and K. Hauser. Commodity telepresence with the avatrina nursebot in the ana avatar xprize semifinals. In RSS 2022 Workshop on "Towards Robot Avatars: Perspectives on the ANA Avatar XPRIZE Competition, 2022.
[57] H. Le, N. Jiang, A. Agarwal, M. Dudik, Y. Yue, and H. Daumé III. Hierarchical imitation and reinforcement learning. In International conference on machine learning, pages 2917-2926.
PMLR, 2018
[58] K. Shiarlis, M. Wulfmeier, S. Salter, S. Whiteson, and I. Posner. Taco: sition via temporal alignment for control. In International Conference on Machine Learning, pages 4654 4663. PMLR, 2018.
Learning task decompo-
[59] Y. Zhu, J. Wong, A. Mandlekar, and R. Martin-Martín. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint urXiv:2009.12290 2020.
[60] A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In Conference on Robot Learning (CoRI.), 2023.
[61] A. S. Morgan, B. Wen, J. Liang, A. Boularias, A. M. Dollar, and K. Bekris. Vision-driven compliant manipulation for reliable, high-precision assembly tasks. RSS. 2021.
[62] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In KDD, 1996.
12

---

[63] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Had- sell, N. de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv: 182.00940, 2018.
[64] C. Wang, R. Wang. A. Mandlekar, L. Fei-Fei, S. Savarese, and D. Xu. Generalization through hand-eye coordination: An action space for learning apatially-ivaarian visuomotor control. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9913-8920, IEEE, 2021.
[65] C. Wang, C. Pérez-D' Arpino, D. Xu, L. Fei-Fei, K. Liu, and S. Savarese. Co-gail: Learning diverse strategies for human-robot collaboration. In Conference on Robot Learning, pages 1279-1290. PMLR, 2022.
[66] C. Wang, L. Fan, Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint 2rX2::3302.1222, 2023.
13

---

Appendix
A
Table of Contents
FAQ (Appendix B): answers to some common questions
Limitations (Appendix C): more thorough list and discussion of HITL-TAMP limitations
Related Work (Appendix D): discussion related work
Tasks (Appendix E): full details on tasks and portions handled by TAMP
Additional Data Throughput Comparisons (Appendix F): additional comparisons on data collection times between HITL-TAMP and conventional teleoperation
Robustness to Pose Error (Appendix G): analysis HITL-TAMP robustness to incorrect object pose estimates
Demonstration Statisties (Appendix H): statistics for collected datasets
Queucing System Analysis (Appendix I): analysis on how the size of the fleet influences data throughput
Additional Details on TAMP-Gated Teleoperation (Appendix J): full details on how
TAMP-gated gated teleoperation works
Policy Training Details (Appendix K): details on how policies were trained from HITL- TAMP datasets with imitation learning
Low-Dim Policy Training Results (Appendix L): full results for agents trained on low-dim
observation spaces (image agents presented in main text)
TAMP Success Analysis (Appendix M): analysis of TAMP success rates and whether policy evaluations could be biased
Additional Details on Conventional Teleoperation System (Appendix N): additional de-
tails on the conventional telcoperation system and why it is representative baseline.
Additional User Study Details (Appendix O): additional details on how the user study was conducted.
14

---

B Frequently Asked Questions (FAQ)
1. How did you select those specific baselines and ablations in Sec. 6?
Our experiments showcase the capabilities of HITL-TAMP as (1) scalable demonstration collection system and (2) an efficient learning and control framework. To show its value in collecting human demonstrations widely-adopted conventional teleoperation paradigm used in prior works that collect and learn from human demonstrations [1, 6, 2, 7, 42, 20, 9, 36, 37, 43, 10, 15, 16, 11] (see
alternative, we compared it extensively against
Table and Fig. 6)
To show its value in learning policies for manipulation tasks, we investigated the value of the core component the TAMP-gated control mechanism (described in Appendix J). We showed that even policies trained on conventional telcoperation data benefit substantially from incorporating the TAMP-gaa -ated control mechanism (Fig. 6). Our TAMP-gare -gted control is a novel control algorithm made possible by key technical components of HITL-TAMP (as described in Sec. 3).
There other systems that are designed for specific contact-rich manipulation (such as peg insertion [44, 45)), but HITL-TAMP was not designed to be specialized for any specific task. Rather, it was meant to be a general-purpose system that can be applied to any contact- rich, long horizon manipulation task, as long as the task can be demonstrated by a human operator, and described in PDDLStream.
2. How does this work compare with other works that combine imitation learning and TAMP?
Prior works, such as [46], trained agents in simulation to imitate demonstration data pro- vided by a TAMP supervisor in simulation. In this way, during deployment, an agent can operate without privileged information (such object poses) required by TAMP. How- ever, this setting makes strong assumption that the TAMP system can already solve the target tasks. By contrast, our work extends TAMP system's capabilities using an agent trained on human demonstration segments collected by HITL-TAMP (training details in Appendix K) in order to solve complex contact-rich tasks in the real world. Training an agent on the TAMP segments collected by HITL-TAMP in order to enable TAMP-free pol- icy deployment is an exciting application for future work. However, it orthogonal the main contributions in this paper.
3. What are the trade-offs between the effort to provide demos and the effort to design models and controllers used TAMP?
Collecting large number of human demos can be labor and time intensive [11, 7, 37], but extensive modeling of a task for TAMP can similarly be time-consuming. Our system achieves good tradeoff, by lessening the modeling burden for TAMP by deferring difficult task segments to the human, and lessening the human operator burden by only asking them to operate small segments of a task. When deploying HITL-TAMP (especially in real- world settings), there significant flexibility in deciding what information available to the TAMP system in order to automate portions of task, and which portions of task should instead be deferred to human operator (or trained agent).
4. How does the TAMP system determine which parts of task plan require a human operator?
We formalize human-teleoperated TAMP skills in Sec. 3.1. While their discrete structure is provided by a human (c.g. which objects are involved), our novel action constraint learn- ing technique (Sec. 3.2) characterizes their continuous action parameters. Human model- ers have flcxibility in deciding which skills should be telcoperated based on the contact- richness and required precision of the interaction. understanding of the TAMP system and the limits of planners and perception determine which parts would require human teleoperation. Other practical alternatives include using uncertainty estimates from perception, or directly applying TAMP to tasks of interest, and
our experiments, we used a prior
15

---

observing sections of failure. Fig. E.1 (in Appendix E) showcases the parts of each task that are handled by the TAMP system and the parts that are handled by the human (or trained agent).
5. What assumptions are needed to apply HITL-TAMP to real-world settings, as op- posed to simulation?
Typically, TAMP systems place high burden real-world perception, accurate percep- tion and dynamics models are often needed by TAMP for planning. Part of the motivation of our work was to reduce this requirement. While we do assume knowledge of crude object models and the ability to associate objects (see Sec. 6.3), we use a very simple perception pipeline in this work. We show that this simple pipeline suffices, even for the challenging Tool Hang task in the real-world since a human or an end-to-end trained policy handles the most challenging, contact-rich interactions. See Appendix G for additional validation that HITL-TAMP can tolerate noisy perception.
6. Why are some of the settings for the real-world Tool Hang task different from the other real-world tasks?
The data collection and policy learning methodology are identical to the other tasks, but there are few minor differences. We used an increased resolution of 240x240 for the RGB images (instead of (20x120) due the need for high-precision manipulation. We also excluded the wrist-view in observations provided the trained agent, since found that completely occluded during the human portions of the task. Finally, evaluated our agent over 25 episodes (instead of 50 evaluation episodes used for the other tasks), because policy evaluation for this task is significantly more time-consuming and obtained task success rate 64%, along with a frame insertion rate of 88%.
7. Why are TAMP plans carried out with joint position controller, while human tele- operation and learned policies use an OSC controller?
Our TAMP system creates plans directly in joint space, so able carry out and track motion plans with higher fidelity by using a joint position controller. On the other hand, human teleoperation requires end effector controller (we use OSC [38]) to provide an intuitive mapping between the user device and robot control. Consequently, we switch between these two controllers depending on whether the TAMP system or the human is operating the robot. See Appendix for more information.
16

---

C Limitations
In this section, we discuss some limitations of HITL-TAMP, which future work can address.
1. Applicable tasks. Our general-purpose system can be deployed on any tasks that (1) can be described in PDDLStream and (2) human operators can demonstrate. We did not engineer the system for any specific task - our system greatly extends the set tasks that can be solved when compared to TAMP alone.
2. Task variety. The tasks in this work are focused on tabletop domains, and there is limited object variety in each task. Scaling HITL-TAMP to work for more scenes and objects requires richer set of assets and scenes (in simulation) and more robust perception pipeline in the real world.
3. Prior information on what is difficult for TAMP. HITL-TAMP requires prior infor- mation (at a high-level) on which task portions will be difficult for TAMP. Being able to automatically identify when human demonstrations needed (e.g. based uncertainty estimates from perception) is left for future work.
4. Perception for TAMP. We assume access coarse object models and approximate pose estimation in order to conduct the TAMP segments. Future work could relax this assump- tion by integrating TAMP methods that do not require object models [35].
17

---

D Related Work
D.1 Demonstration Collection Systems for Robot Manipulation
Recent studies have shown the effectiveness of teaching robots manipulation skills through human demonstration [6, 1, 2, 7, 8, 9]. High-quality,, large-scale demonstrations are crucial to this suc- cess [2]. Although recent advancements have made demonstration collection systems more scalable and user-friendly [6, 36], collecting a substantial amount of high-quality, long-horizon demonstra- tions remains time-consuming and labor-intensive [2]. On the other hand, intervention based sys- tems [47, 43, 48, 49] allow the demonstrator to proactively correct for near-failure cases. How- ever, such systems require users constantly monitor robot task executions, which equally time-consuming and sometimes more pegnitivly-ddmmnddie system uses TAMP-gated mechanism that automatically switches control between the robot and the demonstrator. The mechanism also enables user to demonstrate for multiple sessions asyn- chronously, dramatically increasing the throughput of task demonstration.
than demonstrating task [50]. Our
number of recent works have also investigated automatic control hand-offs in the context of online imitation learning [12, 13, 14, 15, 16, 51, 52, 53]. These works have largely focused on iteratively improving single learned policy, and the gating mechanisms rely predicting task performances and action uncertainties, which are often policy and data-specific. Our work instead proposes to augment a TAMP system with imitation-learne' policies. The symbolic abstractions of the TAMP system readily delineate TAMP's capabilities and can be used to determine the conditions for control hand-offs.
Our HITL-TAMP also acts as TAMP-assisted teleoperation system. However, unlike most prior works in assisted robot telcoperation, for which the aims are for humans to provide high-level guid- ance for low-level autonomous control [54, 55, 56], HITL-TAMP focuses on allowing human teleop- erators to "fill the gap" for a TAMP system to complete goal- gual-directed tasks and enabling the system to become more autonomous by learning skills from the human demonstrations.
D.2 Learning for Task and Motion Planning
Task and Motion Planning (TAMP) is powerful approach for solving challenging manipu- lation tasks by breaking them into smaller, easier to solve symbolic-continuous search prob- lems [5, 22, 4, 23]. However, TAMP requires prior knowledge of skills and environment models, making it unsuitable for contact-rich tasks where hand-defining models is difficult. Recent works have proposed to learn environment dynamic models [24, 25, 26), skill operator models [27, 28], and skill samplers [29, 30]. However, these methods still require a complete set of hand-crafted skills. Closest to our work are LEAGUE [31] and Silver el al. [32] that learn TAMP-compatible skills. However, both works are limited in their real-world applicability. LEAGUE relies on hand-defined TAMP plan sampler and expensive RL procedures learn skills in simulation, while Silver et al. requires hard-coded demonstration policies that already solve the target tasks. Our work instead leverage human demonstrations to both train visuomotor skills and informing TAMP plan sampling. We empirically show that HITL-TAMP can efficiently solve challenging tasks such as making coffee in the real world.
D.3 Imitation Learning from Human Demonstrations
Imitation learning techniques based on deep neural networks have shown remarkable performances in solving real-world manipulation tasks [6, 1, 10, 2, 7, 11]. We take a data-centric view [8, 2, 11] to scaling up imitation learning- HITL-TAMP speeds up demonstration collection for wide range of contact-rich manipulation tasks. A trained HITL-TAMP also acts as hierarchical policy [57]. The key difference to pure data-driven approaches [10, 57, 39, 8, 58] is that in HITL-TAMP, the TAMP framework directly drives the hierarchy to ensure that the learned skills are modular and compatible. Similarly, our work builds on research in combining learned and predefined skills [17, 18, 19, 20, 21] and formalizes human demonstrations and learned skills within TAMP framework.
18

---

E
Tasks
jalm)
Three Plece Assembly (sim)
JAMP
Humen
IAMP
Toog Heng jelm)
TAMP
Humen
TAMP
Human
Coffee (sin)
TAMP
Human
Coffee Full Preperstion (sim)
TAMP
Human
IAMP
Human
LAMP
freal)
Stock Three (rail)
TAMP
Toal Hang (aal)
TAMP
Humen
TAMP
Humen
aill
Figure B.1: Task Segments. We show the human and TAMP segments for each task.
In this section, we present extended task descriptions for each task, including a breakdown which segments the human controls and which TAMP handles (see Fig. E.1).
Stack Three (real). The robot must stack 3 randomly placed cubes. The task consists of 4 total segments - TAMP handles grasping each cube and approaching the stack, and the human handles the placement of the 2 cubes on top of the stack.
Square [59, (sim). The robot must pick nut and place onto peg. The nut is initialized in small region and the peg never moves. This task consists of two segments TAMP grasps the nut and approaches the peg, and the human inserts the nut onto the peg-
Square Broad (sim).: The nut and peg are initialized anywhere on the table.
Coffee [43] (sim real). The robot must pick coffee pod, insert it into coffee machine, and close the lid. The pod starts random location in small, box-shaped region, and the machine is fixed.
The task has two segments TAMP grasps the pod and approaches the machine, and the human inserts the pod and closes the lid.
19

---

Coffee Broad (sim + real). The pod and the coffee machine have significantly larger initialization regions. With 50% probability, the pod placed on the left of the table, and the machine on the right side, or vice-versa. Once side is chosen for each, the machine location and pod location are further randomized in a significant region.
Three Piece Assembly [60] (sim). The robot must assemble structure by inserting piece into base and then placing second piece top of the first. The two pieces are placed around the base, but the base never moves. The tasks consists of four segments - TAMP grasps each piece and approaches the insertion point while the human handles each insertion.
Three Piece Assembly Broad (sim). The pieces are placed anywhere in the workspace.
Tool Hang [1] (sim + real). The robot must insert an L-shaped hook into base piece to assemble frame, and then hang a wrench off of the frame. The L-shaped hook and wrench vary slightly in pose, and the base piece never moves. The task has four segments - TAMP handles grasping the L-shaped hook and the wrench, and approaching the insertion / hang points, while the human handles the insertions.
Tool Hang Broad (sim). All three pieces move in larger regions of the workspace.
Coffee Full Preparation [60] (sim). The robot must place mug onto a coffee machine, retrieve a coffee pod from drawer, insert the pod into the machine, and close the lid. The task has segments first TAMP grasps the mug and approaches the placement location, then the human places the mug on the coffee machine (the placement requires precision due to the arm size and space constraints). Next, TAMP approaches the machine lid, and the human opens the lid (requires extended contact with an articulated mechanism). Then, TAMP approaches the drawer handle, and the human opens the drawer. Finally, TAMP grasps the pod from inside the drawer and approaches the machine, and the human inserts the pod and closes the machine lid.
20

---

Additional Data Throughput Comparisons
Task
HITL-TAMP Time (min)
Conventional Time (min)
Square
13.5
35.0
Square Broad
14.0
48.0
Coffee
22.6
46.4
Coffee Broad
28.8
57.8
Tool Hang
48.0
97.1
Tool Hang Broad
51.5
109.8
Three Piece Assembly
30.0
60.0
Three Piece Assembly Broad
34.9
68.3
Coffee Preparation
78.4
132.7
Total
321.7
655.1
Collection time comparison to conventional teleoperation datasets. An extended comparison of data collection time for 200 demos across several tasks for both HITL-TAMM and the conventional telcoperation system. Some items were estimated using the time spent collecting 10 human demonstrations.
Table F.I:
In this section, we compare how long would have taken collect our 2.1K+ HITL-TAMP demon- strations with conventional teleoperation system. The results are shown in Table F.1. Several of the numbers
estimated by collecting 10 human demonstrations and multiplying by 20 (due the time burden of collecting 200 human demonstrations across all tasks with conventional teleop- eration system). In most cases, HITL- -TAMP takes more than 2x fewer minutes to collect 200 demos than the conventional system.
21

---

G Robustness to Pose Error
Dataset
I.0
1.1
1.2
Square (L.)
100.0 0.0
100.0
99.3 0.9
Square (L2)
100.0 0.0
1.0
100.0 1.0
Coffee (L.I)
100.0 + 0.0
100.0
2.5
Coffee (L2)
100.0 0.0
99.3 0.9
98.0 = 1.6
Table G.1: HITI--TAMP Robustness to Pose Noise. We added uniform pose noise all object poses per- ceived by our TAMP system. We use two levels of uniformly sampled noise I.1 is mm of position noise and 5 degrees of rotation noise, and L2 is 10 mm of position noise and 10 degrees of rotation noise. For each level of noise, we collected 200 demonstrations with our HITL-TAMP system, trained image-based agents on these datasets, and evaluated the agents the LO (no noise). LI noise, and L2 noise setting. The agents only perceive camera images and robot proprioception (i.e. not object poses), and the TAMP system receives noisy object poses. The results show that HITL-TAMP agents retain strong performance.
Since TAMP plans pre-contact poses (constraints learned from human demos), errors in the hand- off location the human operator are completely tolerable, as the human can account for any dif- ferences during their demonstration. Our real-world experiments in Fig. 5 best demonstrate the robustness of our system to pose error. For Coffee, we used an extremely crude box model of the coffee machine without any fine-grained pose registration. For ToolHang, the stand is not accurately captured in the observed point cloud due to the thinness of the stand base and column. Consequently, pose registration is naturally noisy. Despite these problems with perception, we were able to achieve high success rates both tasks with few demonstrations.
In this section, we conduct an additional experiment in simulation to obtain quantitative evidence of HITL--TAMP's robustness to object pose estimation. We first describe our noise model. We added uniform pose noise to all object poses perceived by our TAMP system. We use two levels of uniformly sampled noise L1 is 5 mm of position noise and 5 degrees of rotation noise and 1.2 is 10 mm of position noise and 10 degrees of rotation noise [61]. For each level of noise, we collected 200 demonstrations with our HITL-TAMP system (to be consistent with Fig. 6) on the Square and Coffee tasks, resulting in new datasets in total. We then trained image-based agents on these datasets, and evaluated the agents the LO (no noise), LI noise, and L2 noise setting. We emphasize that the agents only perceive camera images and robot proprioception, not object poses, and the TAMP system receives noisy object poses.
The results are presented in Table G.I. Each row corresponds to agents trained on one of our new datasets and each column corresponds to different levels of noise applied the TAMP system during policy evaluation. Recall that report the success rates across 50 evaluations, where there no TAMP failures, and 3 seeds (discussed further in Appendix K and Appendix M).
When evaluating the agents trained on the L1 and L.2 datasets on the same levels, the results are near-perfect (100% success rate for all except Coffee L2, which gets 98% success), which aligns with the 100% success achieved by our agents on our noise-free datasets (see Fig. 6, left). We also found that training on higher amounts of noise gives our trained agents some level of robustness to lower amounts of noise (e.g. evaluating the L2 models on LO and L2)..
We also analyze the execution failure rate of the TAMP system itself (which corresponds to how often we terminate an episode due to a failed grasp of the nut / coffee pod, or dropping the object in hand). We found that the TAMP system failure rate increases by level: from 0% on L0 to 6% on L1 and 23% on L2 for Square, and from 0% on LO to 6% on LI and 24% on L2 for Coffee. This is to be expected, erroneous poses lead to bad grasp. In such settings where perception errors to cause grasp failures, we could easily have the human teleoperate the grasping part of each trajectory as well during data collection and then have the trained agent learn that task segment as well.
22

---

H Demonstration Statistics
Task
Human
Trajectory (HT)
Trajectory (C)
Square
19.8
582.2
150.8
Square Broad
24.2
647.8
167.9
Coffee
71.6
472.0
199.3
Coffee Broad
90.6
663.7
273.8
Tool Hang
70.4
1297.9
479.8
Tool Hang Broad
71.3
1485.8
522.6
Three Piece Assembly
35.3
897.9
260.1
Three Piece Assembly Broad
39.6
1174.1
342.0
Coffee Preparation
43.8
1328.6
593.2
Stack Three (real)
60.9
499.2
Coffee (real)
295.3
494.9
Coffee Broad (real)
326.5
548.3
Tool Hang (real)
124.3
1144.5
Table H.l: Demonstration Lengths. For each task, report the average length (time steps) of the human segment, average trajectory length our HITL-TAMP datasets (HT), as point comparison, average trajectory length of the conventional system data (C). Note that trajectory contains multiple human segments, we average them.
In Table H.1, we present the average length (time steps) of the human-provided segment, the average trajectory length of our HITL-TAMP datasets (HT), and as point of comparison, the average tra- jectory length of the conventional system data (C). Note that if a trajectory contains multiple human segments, we average across them, and that based on collecting 10 trajectories (the same ones used for the analysis in Appendix F). We that the average human segment is small compared to the entire trajectory length this might help ex- plain the efficacy of our TAMP-gate policy, since the policy is only responsible for short-horizon contact-rich behaviors.
of the conventional system lengths estimates
23

---

I Queueing System Analysis
In Sec. 4 and Fig. 3, we discussed our queueing system, which enables scalable data collection with HITL-TAMP by allowing a single human operator to manage fleet of Nooa robot arms and ensuring that the human operator always kept busy. this section, we provide some additional derivations and analysis how the choice of the number robot arms influences data throughput.
Assuming that the human has an average queue consumption rate (number of task demonstrations completed unit time) of Ry and the TAMP system has an average queue production rate (number of task segments executed successfully per unit time) of Rr, we would like the effective rate of production to match or exceed the rate of consumption.
( 1) 2 RH-
Here, the minus is because robot is controlled by the human. Rearranging, we obtain Nrobux 2 1+ Rx, Thus, the size of the fleet should be at least one more than the ratio between the human rate Re.
of producing demonstration segments and the TAMP rate solving and executing segments.
This number is often limited by either the amount of system resources (in simulation) the avail- ability of hardware (in real world). In practice, human operators also need to take breaks and have an effective "duty cycle" where they are kept busy X% of the time. HITL-TAMP can support this extension as well. Assume that the human is operating the system for Ton and resting for Tor. The human consumes items the queue during Ton at an effective rate of
R
RC (N - 1),
and has the queue filled up during Tar at a rate of (N 1). Ensuring that the human con- sumption rate is less than or equal to the production rate, we have
Ton (RH - -o-RRr(Nee (Nrobox 1)) S Ror (Nvobor - 1).
After rearranging we arrive at
RH X Ri 100'
Nrobot 2 1+
where
X
Too
100
(Tan + Toa)
the human duty cycle ratio.
24

---

J Additional Details on TAMP-Gated Teleoperation
We provide additional details on how TAMP-gated teleoperation works. The TAMP system de- cides when to execute portions task, and when a human operator should complete portion. Each teleoperation episode consists of one or more handoff's where the TAMP system prompts human operator to control portion of task, or where the TAMP system takes control back after determines that the human has completed their segment.
Algorithm I displays the pseudocode of the HITL-TAMP system: TAMP-GATED-CONTET takes as input goal formula G. On each TAMP iteration, it observes the current state s. satisfies the goal, the episode terminates successfully. Otherwise, the TAMP system solves for plan ă using PLAN-TAMP from current state s to the goal G. We implement PLAN-TAMP us- ing the adaptive PDDLStream algorithm [23]. The TAMP system then deploys its controller EXECUTE-JOINT-COMMANN
It
and issues joint position commands the robot to carry planned motions until reaching an action that requires the human. At this time, control switches into tele- operation mode, where the human has full 6-DoF control of the end effector. We use a smartphone interface and map phone pose displacements to end effector displacements, similar to prior tele- operation systems [36, 37, 10]. The robot end effector controlled using an Operational Space Controller [38].
[43], we apply phone pose differences as relative pose commands the cur- rent end effector pose. This allows control be decoupled from the current configuration of the robot arm, which is important the TAMP system prompt the human to takeover in diverse configurations. While the human is controlling the robot, the TAMP system monitors whether the state satisfies the planned action postconditions a. effects. Once satisfied, control switches back to the TAMP system, which replans.
Algorithm TAMP-Gated Teleoperation
1: procedure SAMP-GGTTDD-COTROLLEE
while True
3: 8 4 OBSERVE()
b Estimate or observe state b. State satisfies goal Success! Solve for plan Iterate over actions
4:
if E G then
5:
return True
6:
a < PLAN-TAMP(S, G)
for a E do
8:
if not IS-HUMAN-ACTION(() then EXECUTEE-DINT--CMMAANSS else
9:
10:
while OBSERVE() & a.eff EXECUTE- TELEOP() EXECUTE-TELEOP(
12:
D Teleoperation > Re-observe and re-plan
13:
break
J.1
Example Plan
Consider plan found by the TAMP system for the Tool Hang task on the first planning invocation:
ã [qove[qa, Tı, T1, Pick((fmeee g' Pó qı), move(gi, T2, @2), attach(frame, g P2, Q2 P/ q2, stand), move( Pa, qu), pick(totto p'p qa), move(gs, fu), at tachh gf p4, P4, To. frame)) ).
The values in bold represent constants present in the initial state; the non-bold values are parameter values selected by the planner. The learned preimages enable the TAMP system plan not only trajectory to the first manipulation but also to the second manipulation 72. However, because the third trajectory 73 depends on the resultant configuration z2 planning for it is deferred. Upon successfully achieving At Attached(frmme, stand), replanning produces a new plan.
25

---

K Policy Training Details
this section, we detail how we train policies via imitation learning from the human segments of HITL-TAMP datasets. Many choices are mirrored from Mandlekar et al. [1].
TAMP Teleoperation Control Switch
Training Time
TAMP control
Teleoperation
move (q), T,
Collected Data
TAMP Behavior Cloning Control Switch
Test
Time
move (qI, T, 92)
TAMP control
Learned Policy
Figure K.I: Training and Testing Policies. The top shows the HITL-TAME policy training time, where a human teleoperates certain segments, such as opening coffee machine lid. The bottom row shows HITL-TAMP policy testing time, where the human segments are replaced with a learned policy trained with hehavior cloning using the collected training data.
K.1 Observation Spaces
In our experiments, policies are either trained on low-dim state observations or image observations this kind of flexibility is advantageous eases the burden of perception for deploying TAMP systems in the real world. Low-dim observations include ground-trute object poses, while image observations consist of RGB images from front-view camera and wrist-mounted camera. Both observations include proprioception (end-effector pose and gripper finger width). In simulation, the image resolution 84x84, while in real world tasks, we use resolution of 120x 160 for Stack Three, Coffee, and Coffee Broad, and a resolution of 240x240 for Tool Hang. Our real-world agents are all image-based, since we do not assume that objects can be tracked. The real-world Tool Hang agent did use the wrist-view in observations, since found that was completely occluded during the human portions of the task. The TAMP system only estimates poses the start of each episode. We use simple perception pipeline consisting of RANSAC plane estimation to segment the table from the point cloud, DBSCAN [62] to cluster objects, color- based statistics to associate objects, and Iterative Closest Point (ICP) to estimate object poses. For image-based agents, we apply pixel shift randomization (up to 10% each image dimension) as a data augmentation technique (as in Mandlekar al. [1]).
K.2 Action Space
As described in Sec. 3.3, we collect training data using teleoperation through 6-DoF end-effector control, where an Operational Space Controller [38] interprets delta end-effector actions and con- verts them to joint commands. Thus, the action space for all policy learning is also 6-DoF end- effector poses.
K.3 Training and Evaluation
We use BC-RNN with default hyperparameters from Mandlekar et al. [1] with the exception of an increased learning rate of 10-3 3 for policies trained on low-dim observations, to train policies from
26

---

the human segments in each dataset. We follow the policy evaluation convention from Mandlekar et al. [1], and report the maximum Success Rate (SR) across all checkpoint evaluations over seeds, which is evaluated over 50 rollouts. However, the TAMP system can fail during a rollout. To decouple TAMP failures from policy failures, we keep conducting rollouts for each checkpoint until 50 rollouts with no TAMP failures have been collected, and compute policy success rate over those rollouts (discussion in Appendix M). In the real world, take the final poliey checkpoint from training, and use it for evaluation. Fig. K.1 visualizes the difference between the HITL-TAMP policy at training time, where teleoperation is used, and at testing time, where teleoperation is substituted with learned policy for fully autonomous control.
27

---

L Low-Dim Policy Training Results
Task
Time (min)
SR (im)
TAMP-gated SR (im)
Square (C)
25.0
84.0 + 0.0
91.3 + 5.2
Square (HT)
13.5
100.0 + 0.0
100.0 + 0.0
Square Broad (C)
48.0
88.0 1.6
Square Broad (HT)
14.0
100.0 0.0
100.0 + 0.0
Three Piece Assembly (C)
60.0
55.3 + 0.0
96.0 1 2.8
Three Piece Assembly (HT)
30.0
100.0 0.0
100.0 0.0
Tool Hang (C)
80.0
29.3 + 0.0
60.0 + 19.6
Tool Hang (HT)
48.0
80.7 + 1.9
80.7 + 1.9
Table L.I: Comparison to conventional teleoperation datasets (low-dim). We trained normal and TAMP- gated policies using conventional telcoperation (C) and compared them to HITL-TAMP (HT), TAMP-gatine makes policies trained on the data comparable to HITL-TAMP data, but data collection still involves signifi- cantly higher operator time.
In Table and Sec. 6.2, only presented results with image policies. In this section, show that HITL-TAMP still compares favorably to conventional teleoperation data when trained on low-dim observations. The results are presented in Table L.I.
28

---

M TAMP Success Analysis
Tick
Time (min)
(Jow-dim)
(image)
TAMP (Jow (on-ddim)
SR (Jow (Jow-dim)
TAMP SR
SiR
10010
10000
17.7
11.T4:
Becedl
81,2++27
Coffee
100.0
100.01
100.0 2.0.1
100.0
Coffee Benad
Hong
Hong
43.8++08
Thrse Piece
10:0
Throe Asvambly
Coffre
Table M.1: Analyzing TAMP Success Rates during Policy Evaluations. A more complete set of results from Table on HITL-TAM datasets to demonstrate that policy evaluations do not have significant bias by only evaluating in regions where TAMP is successful. All TAMP success rates are high (above 70%) and most are above 88%.
Recall that when evaluating trained policy, to decouple TAMP failures from policy failures, keep conducting rollouts for each checkpoint until 50 rollouts with no TAMP failures have been collected, and compute policy success rate over those rollouts. In certain cases, this procedure could lead to biased evaluations - for example, if TAMP is only successful for an object in a limited region of the robot workspace. this section, we present the TAMP success rates and raw success rates (including TAMP failures) for the policies in Table 6 (left), and demonstrate that it is unlikely that such bias exists in our evaluations. We present the results in Table M.1 note that the Time and SR columns reproduced from Table 6 (right) for ease of comparison. We see that all TAMP success rates are high (above 70%) and most are above 88%.
29

---

N Additional Details on Conventional Teleoperation System
In this section, we provide additional details on the conventional teleoperation system that we com- pared against in this work (e.g. Table and Fig. 6) as well as explain why it is a representative baseline to compare against. Prior works imitation learning leveraged robot teleoperation systems allow for full 6-DoF control of robot manipulator. These systems typically map the state of teleoperation device, such Virtual Reality controller [6]. 3D mouse [63], smartphone [36, 37], or point-and-click web interface [9], to desired robot end effector pose. They also use an end- effector controller to and achieve the desired pose specified by the teleoperation device. The operator controls the robot arm in real-time by using the teleoperation device.
This teleoperation paradigm has been used extensively in prior work that collects and learns from human demonstrations [1, 6, 7, 42, 20, 9, 36, 37, 43, 10, 15, 16, 11]. In this work, we compared against the Robo'Turk [36, 37] system and smartphone interface, which has been used in several prior imitation learning works [10, 39, 1, 64, 65, 66). It was also used to collect datasets for the robomimic benchmark [1], whose results we also compare against (see Sec. 6.2). This makes it an appropriate baseline. However, is important to note that our HITL-TAMP system is not specific to particular teleoperation interface fact, our system is also compatible with 3D mouse interface [63].
30

---

Additional User Study Details
In this section, we provide additional details on how the user study was conducted. We recruited 15 participants that had varying levels of experience with robot teleoperation: participants were unfamiliar, were somewhat familiar, and were very familiar with it. The purpose of the study was compare our system (HITL-TAMP) to a conventional teleoperation system [36], where task demonstrations were collected without TAMP involvement. Participants underwent brief tutorial (5-10 10 minutes) to familiarize themselves with the smartphone teleoperation interface and to practice collecting task demonstrations using both systems.
Each participant performed task demonstrations on tasks (Coffee, Square (Broad), and Three Piece Assembly (Broad)) for 10 minutes on each system, totaling 60 minutes of data collection across the tasks and systems. To reduce bias, the order of systems task and user (while maintaining the task order). Participants filled out post-study survey rank their experience with both systems. Each participant's 's number of successful demonstrations was recorded to evaluate the data throughput of each system, and agents were trained on each partici- pant's 's demonstrations and across all participants' demonstrations (Sec. 6.1). See Appendix for full details on policy training.
randomized for each
All demonstrations were collected single workstation with an NVIDIA GeForce RTX3090 GPU. We used 6 robot processes (No 6) to ensure that human operators were always kept busy (see Sec. 4).
31


Reviews:
[{"review_id": "O0TZhPmGmq", "reviewer": "Reviewer_LDnh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper aims to address long-term imitation learning tasks that involve rich contact interactions. The authors propose a framework that incorporates human involvement in the learning process to efficiently gather human demonstrations.\n\nThe proposed method divides the execution and learning phases of a given task into two distinct parts. The first part involves minimal physical contact and can be effectively solved using the Task and Motion Planning method. The second part, however, entails contact-rich interactions, necessitating an accurate model and improved sensory precision. As a result, human input becomes crucial in this phase. The authors utilize PDDLStream, a logic-based planning framework, to describe the task's state and the associated contact constraints. Additionally, they employ a hanging task as an example to illustrate how they model the teleoperation actions performed by humans and learn the attachment constraints.\n\n", "summary_of_recommendation": "I recommend this paper as it is well written and technically convincing. The main idea and contribution are clear. A few minor issues and confusing parts make me conservative to give a better score. \n\nDue to my limited knowledge in this field, I would like to see the opinions from the other reviewers to give a more confident recommendation.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Except for the issues to be addressed previously, I have a question:\n\n- Line 238, Why is the current framework robust for demonstrations collected from multiple users? In general, the skills collected from multiple demonstrators will have multi-modalties, thus make the imitation learning in general harder. But in this work, I do not see a clear reason to address this issue.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Feedback on Pros:\n\n- The paper is technically sound and well-written. The main manuscript is comprehensive, and the FAQ part in the appendix addresses most of the questions that arise after reading the main paper.\n- The paper's contribution is based on the recognition that human demonstrations are not needed for all aspects of manipulation. Modern robotics techniques can handle most moving and grasping tasks, while the challenging parts involving contact and friction require human expertise.\n- Building upon this observation, the paper presents a framework that efficiently learns the task and collects human skills.\nThe reported results are reasonable, and the inclusion of videos showcasing both simulation and real robot experiments strengthens the contribution.\n\nFeedback on Cons:\n\n- It would be beneficial to provide a brief description of a conventional teleoperation system in the main manuscript since the main method and discussion rely on this aspect.\n- It seems that the conventional teleoperation system represent a group of baseline methods, as shown in FAQ #1 in appendix, then which specific method was used as the baseline in the experiment and why?\n- The PDDL Stream section on pages 3 and 4 requires significant effort to read and understand. Although it does not significantly impact my comprehension of the paper, condensing this section could improve the overall readability.\n\nMinor Suggestions:\n\n- Line 111: Replace \"p\" with \"g\" to correct the reference to grasp.\n- Consider converting Figure 6 into a table format to align with the text.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "EVqmGWqXpCC", "reviewer": "Reviewer_KfBx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a novel data collection and learning system that combines Task and Motion Planning (TAMP) with human teleoperation. It effectively decomposes a task into easier open space movements which can be done by TAMP, and more contact-rich, complex movements that should be done by human teleoperation. Moreover, there is a queuing system that allows human teleoperator to supervises a fleet of robot to enable greater data efficiency. Extensive human user studies are done on the effectiveness of the system.", "summary_of_recommendation": "I recommend Weak Accept since both the problem is important and execution of the method and the system is solid. However, it would be great to address some of the questions raised above.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. The states for planning seem to require quite a lot of human pre-specification (e.g. Grasp(), Empty(), etc). How well do these predefined states generalize to common robotics tasks? Do a new set of definition need to be specified for each new task?\n\n2. This system requires estimation of object pose information. How sensitive is the system performance with respect to the accuracy of the object pose estimation? I would imagine that the trajectories quality can be degraded if the object pose is wrong, but curious how would that affect performance quantitatively. \n\n2. In 6.3: \"We do not assume any capability to track object poses in real-time. Instead, we allow the human to demonstrate (and the policy to imitate) behaviors from partial observations (RGB cameras)\" -- Could you clarify what this means - when exactly are object poses required?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths of the paper: \n\n1. The problem of enabling efficient data collection is a long-standing and important problem in robotics, and this work that deals with the data collection could be very impactful for future work in fast data collection, which drives robot policy learning on large datasets.\n\n2. The idea of combining TAMP and human teleoperation is novel and useful in two aspects. First, substituting open free space movements with TAMP reduces much human workload of teleoperation and improves efficiency of data collection. Second, it is also a more effective algorithm by combining high-level motor primitives and low-level neural network policy.\n\n3. Extensive study has been done on using the system which involves human operators of different skill levels, which is important for human teleoperation studies.\n\nWeaknesses:\n\n1. The presentation of the method could be clearer - in my understanding, the system composes of two parts: the HITL-TAMP data collection system, and the HITL-TAMP policy. It would be clearer if there could be some more elaborate figure representing the two parts explicitly and the workflow. It is a bit confusing in terms of the presentation as of now.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "B_gMVb9dX4D", "reviewer": "Reviewer_SKPE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work aims at solving long-horizon manipulation tasks by combining the benefits of \"learning from demonstration\" and \"task and motion planning\" approaches and proposes \"Human-In-The-Loop Task and Motion Planning\" (HITL-TAMP), a system that combines automated planning and human control to teach robots complex manipulation skills. \n\nHITL-TAMP allows a human to manage multiple data collection sessions with robots, improving data collection efficiency. The system also learns planning conditions from a few demonstrations, and the learned conditions are used to guide robots in completing tasks. The system learns planning constraints from a few human demonstrations and uses a queuing system to allow a human operator to manage multiple robots. The collected data is then used to train a TAMP-gated policy using an imitation learning framework.\n\nExperiments showed that HITL-TAMP collected more demonstrations and trained proficient agents faster than a conventional teleoperation system.", "summary_of_recommendation": "This paper introduces a novel system, HITL-TAMP, for teaching robots complex manipulation skills, combining the strengths of automated Task and Motion Planning (TAMP) and human demonstrations. \n\nInstead of the innovation in the algorithm, this work proposes a well-designed system for collection data, which is a huge contribution to robot learning community.\n\nThe authors provide a thorough evaluation of the method, including various tasks and real-world testing, showing that HITL-TAMP improves data collection and policy learning efficiency. \n\nTherefore, I recommend accepting this paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In related work, \"intervention-based systems allow the demonstrator to proactively correct for near-failure cases\" misses several relevant papers: \n\n1. Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization\n2. Learning from interventions\n\nSeveral papers studying how to let the neural policy to operate the robot and request the human subjects to provide demonstration are not discussed:\n\n1. Where to add actions in human-in-the-loop reinforcement learning\n2. Ensembledagger: A bayesian approach to safe imitation learning\n3. Learning to share autonomy across repeated interaction\n\nIn Appendix line 536, the paper says \"Human modelers have flexibility in deciding which skills should be teleoperated based on the contact richness and required precision of the interaction.\" Currently, HITL-TAMP requires prior knowledge about which portions of the task will be difficult for TAMP. Can the authors address how this knowledge is obtained and if there are plans to make the system more adaptive?\n\nCould the authors elaborate on challenges faced and potential improvements for real-world application such as autonomous driving? This is important for the task where it is hard to determine which \"portions of the task will be difficult for TAMP\".", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Quality**: The quality of the work appears to be high. The authors present a well-motivated and well-executed study that combines task and motion planning with human demonstrations to teach robots complex manipulation skills. The methods are rigorously defined and the experiments are comprehensive, including a user study, various tasks, and real-world testing.\n\n**Clarity**: The paper is well-structured and clear. The authors provide a detailed introduction to the problem, a clear description of the HITL-TAMP system, and thorough explanations of the experiments and results. Diagrams and examples are used effectively to aid understanding. \n\n**Originality**: The work is original in its approach of combining TAMP with human demonstrations to efficiently collect data for robot training. The proposed system, HITL-TAMP, is a novel contribution to the field of robot learning, providing a solution to the challenge of teaching robots complex, long-horizon tasks.\n\n**Significance**: This work is significant as it addresses a key challenge in robotics: teaching robots to perform complex tasks. The method proposed could potentially be applied in various fields where robotic manipulation is used, such as manufacturing or service robotics.\n\nStrengths:\n\n1. The work provides a novel and effective solution for teaching robots complex manipulation skills.\n2. The proposed method combines the strengths of automated planning and human demonstrations, achieving efficient data collection and policy learning.\n3. The authors provide a thorough evaluation of the method, including a user study, various tasks, and real-world testing.\n4. The method enables a single human operator to manage a fleet of robots, improving scalability.\n\nWeaknesses:\n1. The current HITL-TAMP system requires prior knowledge about which task portions will be difficult for TAMP, which could limit its applicability.\n2. The system assumes access to coarse object models and approximate pose estimation for real-world applications, which might not always be available.\n3. While the authors show that HITL-TAMP can train proficient agents quickly, the long-term adaptability and robustness of these agents in changing or unpredictable environments is not discussed. \n\nOverall, this paper makes contribution to the field of robotic learning. The authors provide a well-motivated, clear, and comprehensive study that advances our understanding of how to efficiently teach robots to perform complex tasks.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "0_cDff8prj", "reviewer": "Reviewer_usXb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes to use teleoperated actions within task and motion planning. These non-autonomous actions are specified in the same way as the autonomous actions, namely by specifiying constraints, preconditions and postconditions for PDDL. Following the procedure of Wang et al. [26], some of the constraints can also be learned from teleoperation demonstrations. Using these teleoperation actions, multiple robots can be controlled by a single human, using PDDLStream to plan the next action and putting the robot in a queue, whenever a teleoperation action was chosen. Furthermore, the demonstrations aquired during teleoperation are used to learn a policy using behavioral cloning, such that eventually human teleoperation is no longer required. This paperline is evaluated in a user study showing it can increase the effectiveness of teleoperation and that the learned policies can achieve success rate on long-horizon tasks.", "summary_of_recommendation": "The overall idea is reasonable, but the submission lacks a clear contribution. Furthermore, the whole pipeline needs to be presented in more detail.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Please provide more details on the constraint learning.\n\n- What were the state-action spaces for the different experiments?\n\n- (Minor) Line 96: \"AtGrasp(o,p)\" and Line 111 \"grasp p\": Should it be \"g\" instead of \"p\" in both cases?\n\n- The learned \"constraints\" sound more like preconditions for successfully executing the task. Can you please explain the difference?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "(+) It is reasonable to combine TAMP and teleoperation to solve long-horizon tasks that contain difficult segments that require teleoperation. It also makes sense to use these demonstrations to learn policies that avoid the need for teleoperation.\n\n(+) I am also not aware of prior work that used a similar approach to combine TAMP and teleoperation.\n\n(-) However, the contribution seems to be quite small. It is not clear to me what would be the main takeaway: That we can use teleoperation within task and motion planning? That we can control multiple robots in parallel by putting them into a queue? That we can use the demonstrations for imitation learning? All of this seems to be quite obvious. The specific pipeline could be an important contribution, if it was less limited. The overall idea is implemented rather naively, for example, by using simple behavioral cloning for learning the policies. As a consequence, it seems hard to believe that the pipeline could be scaled to real applications without substantial changes. \n\n(-) Another issue with the current submission is that the approach is not sufficiently well explained. While the writing is good and easy to comprehend, many important details are not at all described. For example, for learning the constraint the paper only states that the constraints are modeled as distributions and how the data set is collected, but the paper does not state how the distributions are modeled and optimized. While I assume, that the proposed method closely follows the prior work [26], that would need to be stated more explicitly, and also the approach should be stated in a bit more detailed so that the reader is not completely lost without consulting the prior work. Also, important definitions are missing. What is the grasp g? Is it an SE-3 pose? The paper states that \"forward kinematics f rom configuration q multiplied with grasp p produces pose p\". Does this refer to a multiplication of homogenous transformation matrices? That would mean that the grasp is specified relative to a given end-effector pose? Furthermore, the experimental setup needs to be explained in much more detailed. How was the user study conducted (What instructions where given, which setting was tested first, what exactly where the state-action spaces in each experiment?)", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors' responses to the reviewer's comments have been incorporated into the meta review. The paper introduces a framework that integrates teleoperated actions with task and motion planning (TAMP) for addressing long-horizon tasks containing challenging segments that require human intervention. The proposed approach combines TAMP and teleoperation to effectively switch between autonomous and human-guided actions, with the goal of learning policies that reduce the reliance on teleoperation over time.
Strengths:
1.	The reviewers recognize the rationality of the approach, merging TAMP and teleoperation to solve complex tasks that involve both autonomous and human-guided actions.
2.	The integration of learned constraints, TAMP-based actions, and teleoperated actions to improve the efficiency and autonomy of the system is appreciated.
3.	The evaluation of the approach through a user study and the demonstration of improved teleoperation effectiveness, along with the policy learning for autonomous execution, are highlighted as valuable contributions.
Weaknesses:
1.	The contribution of the paper is perceived as small, and the main takeaways are not clearly highlighted.
2.	The implementation of the proposed pipeline is considered to be relatively basic, and its scalability to real applications is questioned.
3.	The presentation of the approach and important details is deemed insufficient.
The reviewers acknowledge the novel combination of TAMP and teleoperation, but they express concerns about the clarity of the main contributions and the limited scope of the implementation. They note that while the idea of the approach is reasonable, the paper needs to better define its contribution and explain the specific methodology. Additionally, the reviewers request further elaboration on the constraint learning process, the experimental setup, and the definitions of key terms.
In response, the authors address these concerns by emphasizing the contributions in the introduction and clarifying the unique aspects of their approach. They explain the methodology more explicitly, particularly regarding constraint learning, and provide additional details about the experimental setup. Reviewers now revised recommendations to accept.
Decision: Accept (Poster)