SPIRE: Synergistic Planning, Imitation, and Reinforcement Learning for Long-Horizon Manipulation

Authors: zihan  zhou
animesh  garg
dieter  fox
caelan  garrett
ajay  mandlekar
  nvidia
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 12:26:38

Abstract: Robot learning has proven to be a general and effective technique for programming manipulators. Imitation learning is able to teach robots solely from human demonstrations but is bottlenecked by the capabilities of the demonstrations. Reinforcement learning uses exploration to discover better behaviors; however, the space of possible improvements can be too large to start from scratch. And for both techniques, the learning difficulty increases proportional to the length of the manipulation task. Accounting for this, we propose SPIRE, a system that first uses Task and Motion Planning (TAMP) to decompose tasks into smaller learning subproblems and second combines imitation and reinforcement learning to maximize their strengths. We develop novel strategies to train learning agents when deployed in the context of a planning system. We evaluate SPIRE on a suite of long-horizon and contact-rich robot manipulation problems. We find that SPIRE outperforms prior approaches that integrate imitation learning, reinforcement learning, and planning by 35% to 50% in average task performance, is 6 times more data efficient in the number of human demonstrations needed to train proficient agents, and learns to complete tasks nearly twice as efficiently. View https://sites.google.com/view/spire-corl-2024 for more details.

Main paper: 
 introduction: 
 Reinforcement Learning (RL) is a powerful tool that has been widely deployed to solve robot manipulation tasks [1, 2, 3, 4] . The RL trial-and-error process allows an agent to automatically discover solutions to a task and improve its behavior over time. However, in practice, it often relies on careful reward engineering to guide the exploration process [5, 6] . The exploration burden and reward engineering problem is even more challenging to overcome for long-horizon tasks, where an agent must complete several subtasks in sequence in order to solve the task [7] .Imitation Learning (IL) from human demonstrations [8, 9] is a popular alternative to reinforcement learning. Here, humans teleoperate robot arms to collect task demonstrations. Then, policies are trained using the data. This alleviates the burden of reward engineering, since correct behaviors are directly specified through demonstrations. This paradigm has recently been scaled up by collecting large datasets with teams of human operators and robots and shown to be effective for different real-world manipulation tasks [10, 11, 12] . While these agents can be effective, they typically are imperfect, with respect to both success rates and control cost, and not robust to different deployment conditions, especially when it comes to long-horizon tasks [13] .One way to integrate the benefits of both IL and RL is to first train an agent with IL and then finetune it with RL. This can help improve the IL agent and make it robust through trial-and-error , while also alleviating the need for reward engineering due to the presence of the demonstrations. Several works have used this paradigm successfully, but long-horizon manipulation still remains challenging due to the burden of exploration and long-term credit assignment [7] . (Left) SPIRE first attempts to solve the task with a TAMP system. When the TAMP planner encounters an action deemed too hard to plan, it then enters the handoff section and delegates the action to a human teleoperator to manually complete it. We record the trajectories from the human operators to build a demonstration dataset and train an IL policy with it. Finally, we train an RL policy to fine-tune the IL policy via warmstarting and deviation constraining. (Right) The four handoff sections in Coffee Preparation.One effective approach for long-horizon manipulation is to leverage a hybrid control paradigm, where the agent is only responsible for local manipulation skills, instead of the full task [14, 15] . An example is the HITL-TAMP system [14] , where an agent is trained with IL on small contactrich segments of each tasks, and the rest of the task is performed using Task and Motion Planning (TAMP) [16] . Another related approach is PSL [15] , which learns an agent using RL instead of IL on small segments, and uses motion planning to sequence the learned skills together. These approaches are effective for challenging long-horizon manipulation tasks, but they still often do not train perfect policies, suffering from some of the pitfalls of IL and RL. In this paper, we take inspiration from these approaches and create a hybrid control learning framework that allows for efficient imitation learning and RL-based finetuning of agents to address long-horizon manipulation tasks.We introduce Synergistic Planning Imitation and REinforcement (SPIRE), a system for solving challenging long-horizon manipulation tasks through efficient imitation learning and RL-based finetuning. SPIRE decomposes each task into local manipulation segments that are learned with a policy and segments handled by a TAMP system. The manipulation segments are first trained via imitation learning and then finetuned with reinforcement learning. Our approach on 9 challenging manipulation tasks reaches an average success rate of 87.8%, vastly outperforms TAMP-gated IL [14] (52.9%) and RL [15] (37.6%). In the subset of tasks where SPIRE and IL both reach a high success rate, our method only uses 59% of the steps required by IL to complete the task. In Tool Hang, SPIRE fine-tunes an IL policy with only 10% success rate to 94%. We perform a thorough analysis of our method and also show that in many cases, a handful of demonstrations suffice for learning challenging tasks. Compared with IL, SPIRE improves the overall demo efficiency by 5.8 times in the evaluated subset of tasks.• We propose SPIRE, a hybrid learning-planning system that synergistically integrates the strengths of behavior cloning, reinforcement learning, and manipulation planning. SPIRE first learns a TAMP-gated policy with BC and then improves it with RL.• We introduce key insights to enable RL-based finetuning with sparse rewards in this regime, including a mechanism to warmstart the RL process using the trained BC policy, a way to constrain exploration to be close to the BC agent behavior, and a multi-worker TAMP framework to optimize the throughput of SPIRE's RL process.• We evaluate SPIRE on a suite of long-horizon contact-rich tasks and find that it outperforms prior hybrid learning-planning approaches success rate averaged across tasks (87.8%, compared to 52.9% and 37.6%), execution efficiency (episodes are only 59% the length of the BC agent), and human demonstration efficiency (6 times less data required than BC to train similar agents). 
related work: 
 Hierarchical approaches for long-horizon tasks. Hierarchical approaches decompose the challenging long-horizon tasks into easier-to-solve subtasks. RL based methods explore the division of sub-tasks with reusable skills [17, 18, 19, 20, 21] . [22, 23, 24, 25, 26] build hierarchical RL solutions with subpolicies and metacontrollers. Our work instead leverages a planner that provides guidance on which policies to learn as well as initial and terminal state distributions of tasks, compared to bottom-up HiRL methods, which tend to be data inefficient. Notably this top-down breakdown may also be achieved with a Language Model which can provide a plan composed of steps and sub-goal targets [27, 28, 29, 30, 31, 15] .Robot manipulation with demonstrations. Behavior cloning (BC) [32] learns a policy by directly mapping observations to actions and is typically trained end-to-end using pre-collected pairs of observation and behavior data. While this is seemingly a supervised learning problem, the context of robotics adds challenges. BC datasets tend to contain data sampled from multimodal distributions, due to intra-expert variations. Recent work address this problem with implicit models including those derived from energy-based models [33, 34] , diffusion models [35, 36, 37, 38, 39] , and transformer models [40, 41, 42] . Another challenge is the correlation in sequential data, which can lead to policies which are susceptible to temporally correlated confounders [43] . Recently several works have set out to handle this by predicting action chunks. For example, the Action Chunking Transformer (ACT) line of work [44, 45] and diffusion policy [35] . While BC-based methods combined with high-capacity models enable complex robotics tasks from demonstrations, problems in robustness and long-horizon generalization remain.Experts and their demonstrations can be used in combination with RL in multiple ways, including acting as task specifications, exploration guidance, and data augmentation. Inverse RL [46, 47, 48] learns a reward model for RL from demonstrations; [49, 50, 51] explore using demonstrations to bootstrap the reinforcement learning process; [52] uses state matching for reward computation in RL. [53] shares a similar setup with ours, where they also warmstart RL with a BC policy and use a masked BC loss the constrain the RL policy from deviating. [54, 55] propose to fine-tune a semiexpert initial policy by training a residual policy on top of it with RL. Different from the mentioned works, we focus on multi-stage robotic manipulation tasks with high-dimensional input spaces.Learning for Task and Motion Planning. Task and Motion Planning (TAMP) is a search and planning-based approach for sequential manipulation tasks [16, 56, 57, 58] . However, pure TAMPbased methods suffer from reliance on accurate environment modeling and prior knowledge of the skills, making those methods less suitable for tasks involving complex contact-rich skills. 
method: 
 Our approach Synergistic Planning Imitation and REinforcement (SPIRE) learns and deploys closed-loop visuomotor skills within a TAMP system (see Fig. 1 ). First we frame our problem as a policy learning problem across a sequence of Markov Decision Processes (Sec. 3.1). Next, we describe our approach for incorporating both classical and learned robot skills into TAMP (Sec. 3.2) to enable TAMP-gated learning. Next, we describe how we train an initial agent with TAMP-gated Behavioral Cloning (BC) (Sec. 3.3). We then propose an RL-based finetuning algorithm to improve the BC agent with RL (Sec. 3.4). Finally, we introduce a parallelized training scheduler that is able to intelligently manage dependencies among stages when conducting RL in our setting (Sec. 3.5).In our setup, each robot manipulation task can be decomposed into a series of alternating TAMP sections and handoff sections, where TAMP delegates control to a trained agent π. These sections are TAMP-gated [14] , as they are chosen at the discretion of the TAMP system, and typically involve skills that are difficult to automate with model-based planning. We wish to train an agent π to complete these handoff sections efficiently and reliably. We model our TAMP-gated policy learning problem as a series of Markov Decision Processes (MDPs), M := (S, A, T, {r i }, {p i 0 }, γ) N i=1 , where N is the number of MDPs (each corresponds to a TAMP handoff section), S and A are the state and action space, T is the transition dynamics, r i (s) and p i 0 are the i-th reward function and initial state distribution, and γ is the discount factor. The start and end of each handoff section is chosen by TAMP, consequently, TAMP determines the initial state distribution p i 0 for each handoff section, and the reward function r i (s), which is a sparse 0-1 success reward based on the successful completion of the section. Our goal is to train a stochastic policy π : S → ∆ A that maximizes the expected return J(π) := E (i,τ )∼π t=0 γ t r i t . We next describe the TAMP system.Task and Motion Planning (TAMP) [16] is a model-based approach for synthesizing long-horizon robot behavior. TAMP integrates discrete (symbolic) planning with continuous (motion) planning to plan hybrid discrete-continuous manipulation actions. Essential to TAMP is a model of the actions that a planner can apply and how these actions modify the current state. From such a model, TAMP solvers can search over the space of plans to find a sequence of actions and their associated parameters that satisfies the task.In SPIRE, we seek to implicitly learn a select set of TAMP actions that are impractical to manually model and then combine them with traditional actions through planning. In essence, our strategy is to learn policies π that control the system from TAMP precondition states to postcondition states (typically described by effects). We adopt the modeling strategy introduced by Mandlekar et al. [14] and deploy PDDLStream [58] to solve each TAMP problem. See Appendix G for a summary of our planning model. Fig. 2 if s ∈ G thenreturn True⃗ a ← PLAN-TAMP(s, G)for a ∈ ⃗ a do 8:if a.type = "RL" thenπ ← a.policy TAMP-Gated Data Collection. We collect an initial dataset of human demonstrations through TAMP-gated human teleoperation, where the human operator collects demonstrations for handoff sections when prompted by TAMP, to form the demonstration dataset D = {{(s t , a t ) Hi t=1 , g i }}, where s t ∈ S, a t ∈ A and H i is the horizon, and g i is the handoff section of the i-th trajectory. To improve the data collection efficiency, we replicate the task queuing system from [14] .Given the dataset D, we train a Behavioral Cloning (BC) policy parameterized by ϕ to minimize the negative log-likelihood loss over the demonstration dataset:ϕ * = argmin ϕ E (s,a)∼D [-log π ϕ (a|s)].The trained BC agent π ϕ may have substantial room for improvement, depending on the complexity of the task, and the number of demonstrations available for training. We next describe our RL-based finetuning procedure (Sec. 3.4) that allows this agent to be improved through reinforcement learning.Given a trained BC agent π ϕ , we wish to train an RL agent π θ to improve performance further. To avoid reward engineering, we only assume access to sparse 0-1 completion rewards for each handoff section provided by TAMP (Sec. 3.1). However, exploration in sparse-reward settings has been shown to be challenging [59, 60, 61, 62] , especially in continuous state and action spaces. Fortunately, we can use the BC policy trained in the previous section as a reference point for exploration -we want to restrict the behavior of the RL policy to be in a neighborhood of the BC policy. This is achieved by a) warmstarting the RL policy optimization using the BC policy, and b) enforcing a constraint on the deviation between the RL policy and the BC policy. Warmstarting RL optimization with BC. We tested two ways to warmstart the RL agent. Initialization. One method is to initialize the weights of the RL agent with those of the trained BC agent, θ ← ϕ * , where ϕ * = arg min ϕ L BC (ϕ), and subsequently finetune the weights with online RL objectives. Despite being easy to implement, this can be less flexible since it requires the agent structure of the RL and BC policies to match. Furthermore, researchers have found that retraining neural networks with different objectives can cause the network to lose plasticity [63] , which can make the policy harder to optimize because of the objective shift from BC to RL. Residual Policy.An alternative way is to fix the BC policy as a reference policy and train a residual policy on top of it. Let the residual policy be π + θ (s). The residual policy shares the same action space as the normal policy but is initialized to close to zero. The final action is defined as a summation of the reference action a ∼ π ϕ * (s) and the residual action a + ∼ π + θ (s). In practice, we only add the mean of the reference policy to the residual action instead of sampling the reference action.Constraining Deviation between BC and RL agents. The sparsity of reward signals produces high-variance optimization objectives, which can lead the RL policy to quickly drift away from BC and lose the exploration bonus from warmstarting. Therefore, it is critical to constrain the policy output to be close to the BC agent throughout the training process. We achieve this by imposing a KL-divergence penalty. We conclude our RL optimization objective as follows: J F T (θ) := J(π θ )-αD KL (π θ ∥π ϕ * ), where D KL (p∥q) := E (s,a)∼p log p(a|s) q(a|s) and α is the weight for the penalty term.Making our TAMP-gated framework compatible with modern reinforcement learning procedures requires addressing several challenges. First, TAMP can take dozens of seconds for a single rollout, which severely lowers the throughput of RL exploration. Second, the TAMP pipeline executes each section sequentially, which means that later handoff segments can only be sampled when previous handoff segments are completed successfully. This leads to an imbalance of episodes for the different handoff segments and is potentially problematic for the RL agent. We propose a multi-worker TAMP scheduling framework to integrate TAMP into RL fine-tuning. The framework consists of three components -a group of TAMP workers that run planning in parallel, a status pool that stores the progress of the workers, and a scheduler that distributes tasks to the workers and balances the initial states. We further describe how the framework allows for curriculum learning, and how the framework accelerates learning efficiency for RL training. See Appendix H for more details. the scheduler. Depending on the command, the worker either resets itself or starts interacting with the environment by exchanging actions and states with the scheduler. If the current section is solved, the worker sends a success signal to the scheduler and runs TAMP until the next handoff section.The scheduler is a centralized component that manages the TAMP workers. It also provides an environment abstraction to the single-threaded RL process. The scheduler is configured with a sampling strategy. Upon initialization, it first pops an item from the status queue.According to the sampling strategy, the scheduler either rejects this section, in which case it sends a resetting signal to the corresponding worker; or starts a new episode and interacts with the worker.Curriculum Learning. The behavior of the scheduler depends on a sampling strategy, allowing it to function as a curriculum for the RL agent. We consider two strategies: permissive is the default strategy that allows all sections through, while sequential only accepts a section when the success rate of passing all the previous sections reaches a threshold. sequential allows controlling the initial state distribution during the early stages of training, to ensure the RL agent achieves proficience in each section sequentially before continuing onto the next section.Remarks on Efficiency. Suppose a TAMP planning process takes at most T seconds over the episode; each environment interaction step, counting communication latency, takes at least t seconds; and each handoff section is at least H steps. If the number of TAMP workers n ≥ T tH , the proposed multi-worker TAMP scheduling framework reaches a throughput of at least 1/t frames per second. In comparison, the single-worker counterpart has a worst-case throughput of H T +tH frames per second. Suppose that the planning process is slower than the handoff sections by a factor k (e.g. T = k • tH), then our framework is faster than the single-worker alternative by a factor of k + 1. 
experiments: 
 Tasks. For evaluation, we follow [14] and choose a set of long-horizon manipulation tasks, namely Square, Coffee, Three Piece, and Tool Hang. We also include the broad variants of those tasks, where we use a broad object initialization region, and Coffee Preparation, which has the longest horizon with four handoff sections. See Appendix E for more details.Environment Details. Observation space. For most tasks, we use a single 84 × 84 RGB image from the wrist-view camera. For Tool Hang, we use the front-view camera instead since the wristview is mostly occluded. For Tool Hang Broad and Coffee Preparation, we use both wrist-view and front-view cameras, as well as proprioception state (end-effector pose and gripper finger width). Action space. Actions are 7-dimensional (3-dim delta end-effector position, 3-dim delta end-effector rotation, 1-dim gripper actuation). Horizon. Each handoff section is limited to 100 steps (5 seconds with 20Hz control frequency) for all tasks, except for Tool Hang Broad, where the limit is 200 steps.Baselines. We compare our method with two baselines: HITL-TAMP-BC (BC), which is adapted from [14] to match our network structure; and TAMP-gated Plan-Seq-Learn (RL), which is adapted from [15] by replacing the LLM-based planning system with our TAMP system for fair comparison. We collected 200 human demonstrations for each task to train the behavior cloning policy. For RL, we use DrQ-v2 [64] as the base algorithm. See Appendix B for more details.Evaluation. We evaluate each trained agent for 50 rollouts and report the success rate and average completion steps in the successful rollouts. We train 5 seeds for each algorithm and report the bestperforming agent (success rate-wise, tie-breaking with average steps) unless otherwise specified.SPIRE outperforms both TAMP-gated BC and RL. We compare our method with the TAMPgated BC [14] and RL [15] baselines across all 9 tasks (see Fig. 3 ). SPIRE reaches 80% success rate in 8 out of 9 tasks, while BC and RL only reach 80% in 3 tasks each. In Tool Hang, our method reaches 94% success rate despite the BC counterpart only having 10%, which is over 9-times improvement. Remarkably, this low-performing BC agent is enough to help address the exploration burden (unlike RL, 0% success) and train a near-perfect agent. Across all 9 tasks, SPIRE averages a 87.8% success rate, while BC and RL only average 52.9% and 37.6% respectively. SPIRE produces more efficient agents than BC through RL fine-tuning. SPIRE agents have lower average completion times than their BC and RL counterparts (Fig. 3 , right). Even in tasks such as Square, Square Broad, Coffee, Three Piece, where BC policies already have high success rates, our method improves the efficiency by only using an average of 59% completion time. SPIRE's use of the BC agent helps address the RL exploration burden on challenging longhorizon tasks. Exploration in RL with sparse rewards is extremely challenging, especially for robot manipulation tasks for their continuous and high-dimensional observation and action space. Our method solves the initial exploration problem by anchoring policy learning around the BC agent. As shown in Figure 3 , RL policies without utilizing BC only reach nonzero success rates in Square, Coffee and their Broad variants, all of which have only one handoff section and relatively shorter horizons. Even in Coffee Broad, RL encounters exploration difficulties due to the broader object distribution, resulting in only partially solving the task.Qualitatively, SPIRE can improve agent behavior without introducing undesirable behavior, unlike RL. Safety awareness has always been a critical matter in robotics learning. Safety constraints can be hard to define with numerical values, which adds to the challenges of realizing safety in RL. We notice that in Coffee, RL policy has a much shorter completion time than our method. This is at the cost of ignoring safety concerns. We compare two rollouts of RL and our method in Figure 4 . The RL-trained policy attempts to close the lid by knocking the coffee machine with the arm, which can potentially damage the robot and the coffee machine and even cause danger to humans; while our method preserves safety awareness by following the demonstration's practice of closing the lid with its fingers. SPIRE can train proficient agents using just a handful of human demonstrations. BC methods can require several human demonstrations to train proficient agents, which can be a major drawback due to the cost of collecting this data [9] . We reduce the number of human demonstrations used by SPIRE to 10 and 50 (instead of 200 as in Fig. 3 ), and we plot the minimum of demonstrations needed to reach at least 80% success rate in Fig. 5 . As the plot shows, SPIRE can successfully fine-tune a BC policy trained with as few as 10 demos in all evaluated tasks except for Tool Hang and Coffee Broad, for which 50 demos are enough. In the 7 tasks, our method needs 150 demos in total, while BC needs more than 870, a 5.8× improvement in efficiency.We conduct two ablative studies to investigate (1) the value of the KL-divergence penalty and (2) the value of curriculum learning, governed by the two scheduler sampling strategies permissive and sequential. In this section, we compare the performance distribution of the 5 runs instead of only the top-1 run for a more comprehensive evaluation.Value of divergence penalty. We ablate the divergence penalty on two representative tasks, Three Piece & Tool Hang (Table 6 ) and observe a drastic performance drop (84% to 17.6%, 74% to 0%). Mean and standard deviation (in parenthesis) of success rates across the 5 seeds in Three Piece and Tool Hang, with and without KL-divergence penalty. In both tasks, the divergence penalty improves the performance by a large margin.The sparsity in rewards leads to high-variance optimization objectives for RL. As a result, even when warmstarted with BC, the RL policy can quickly deviate from it, especially when the chance of reaching the reward signal is low. Therefore, constraining the policy close to BC throughout the training is critical. We select two representative tasks, Three Piece and Tool Hang for this ablation. The result is shown in Figure 6 . Without the divergence penalty, the RL policies deviated immediately and never returned.Value of curriculum learning. We compare the two sampling strategies in Tool Hang task. The result is shown in Figure 5 . sequential strategy shows a much smaller variance, while permissive produces the better top-1 seed performance. The main difference between the two strategies is how the second section states emerge during training. For permissive, the second section states emerge gradually as the success rate of passing the first section gets higher, resulting in a more gentle distributional shift that leads to a higher overall success rate; for sequential, the shift is more abrupt, but it has fewer distraction states in the early stage, resulting in a more stable training process. 
conclusion: 
 We presented SPIRE, an integrated approach for deploying RL, BC, and planning harmoniously. We showed how BC can be used to not only warm-start RL but also guide the RL process via focused exploration. We introduced a scheduling mechanism to improve RL data throughput and increase learning efficiency. Finally, we evaluated SPIRE in simulation against recent hybrid learningplanning baselines and found that SPIRE results in more successful and efficient policies.Limitations. We focus on tasks that center around object-centric manipulation of rigid objects in table-top environments. We assume that a human teleoperator can demonstrate the learned skills to warmstart RL. The TAMP component assumes that the state is observable and comprised of rigid objects, possibly connected with articulation. To simplify RL training, we only considered Markovian policies; however, using neural network architectures with history, such as RNNs, may boost performance [9] . Finetuning BC policy with RL requires the simulation to be efficient. 
 
 Appendix: The Appendix contains the following content. Hyperparameters. The base RL algorithm for all our experiments is DrQ-v2 [64] . The specific hyperparameters are in Table 1 .Observation. For most tasks, we use one 84 × 84 RGB image from the wrist camera as the only observation. For Tool Hang, we use a front-view camera instead since the wrist-view is heavily occluded. For Tool Hang Broad and Coffee Preparation, we use both camera views plus proprioception state (end-effector pose and gripper finger width). We use the default CNN structure from DrQ-v2 to encode the image observations. For tasks with multiple observations, we first encode the image observations each with an independent CNN network, then concatenate the CNN outputs alongside the low-dimensional observations such as proprioception states to form the feature vector.Action. All of our tasks share a 7-dimensional continuous action space. It is models 6-DOF delta movement of the end-effector along with 1 dimension for finger control. The action is modeled as a normal distribution with a scheduled standard deviation.We provide an additional ablation study on the high-level planner, TAMP. To do so, we treat the whole task as one handoff section. The agent only receives a reward of one if it completes the whole task. We collect 200 full demonstrations in Square, train a BC policy, and apply SPIRE to finetune the BC policy. Since the trajectory becomes longer and the robot now needs to handle object transportation, a single local wrist-view becomes insufficient. We thus include both the wrist view and the global front view, as well as the robot proprioception states in the observation for the w/o TAMP variant. The result is shown in Table 2 . Even though the w/o TAMP variant has more information from observations, the BC and RL policies are significantly worse than the w/ TAMP counterpart. The increased horizon makes the BC policy easier to drift away to regions less frequently visited in demonstrations and makes RL exploration much harder. In Square, despite the low starting quality, SPIRE still fine-tunes BC to reach a 94% success rate, demonstrating the effectiveness of RL fine-tuning. However, when the initialization range increases in Square Broad, even SPIRE fails to find an acceptable policy.In conclusion, TAMP (1) confines the agent-controlled section to a small local area, reducing the need for global information, and (2) decreases the horizon (11.6 w/ TAMP, 101.7 w/o TAMP in Square) for the learned agent, reducing compounding errors and exploration difficulty.In each handoff section from TAMP, SPIRE utilizes the demonstrations by training a behavior cloning agent and using RL to fine-tune it. There are alternative methods to combine expert demonstrations and RL, which can be readily plugged in as replacements to SPIRE. In this section, we make connections from our method to GAIL [46] . The discriminator-based IRL reward in GAIL serves the same purpose as our KL penalty term -preventing the current policy from deviating from the expert policy. We draw further connection by showing that our KL penalty is the same as the IRL reward function in GAIL with an alternative discriminator objective and a different reward form.Let π E be the expert policy. The IRL reward function in GAIL is -log(1 -D(s, a)), where D : S × A → [0, 1] is the discriminator that maximizesJ(D) := E τ ∼π [log(1 -D(s, a))] + E τ ∼π E [log(D(s, a))]If we use an alternative objective:Ĵ(D) := E s∼π E ,a∼Unif [-D(s, a)] + E τ ∼π E [log(D(s, a))]The alternative objective discriminates π E from a fixed policy rather than the current learned policy π. Assume π E has full support, then maximizing Ĵ(D) is equivalent to maximize for every s ∈ S:EQUATIONEQUATIONEQUATIONEQUATIONEQUATION)where H is the entropy. ( 5) holds since log x ≤ x -1 for all x > 0, and only equates when x = 1, i.e., D(s, a) = π E (a | s). Since ( 7) is a constant, the maximum of Ĵ(D) can be taken when (5) equates, which means the optimal solution of Ĵ(D) is D(s, a) = π E (a | s). Our KL penalty then is equivalent to using an IRL reward of log( D(s, a)) = log π E (a | s). Square and Square Broad. The robot must pick up a nut and place it onto a peg. This task has 1 handoff section, where the learned agent places the nut. The Broad version increases the initialization range of both the nut and the peg.Three Piece and Three Piece Broad. The robot must insert one piece into a base and place another piece on top of the first. This task has 2 handoff sections, where the learned agent places the two pieces. The Broad version increases the initialization range of all three pieces including the base.Tool Hang and Tool Hang Broad. The robot must first insert a L-shaped piece into a base to assemble a frame, then hang a wrench off of the frame. This task has 2 handoff sections, where the learned agent inserts the L-shaped piece and hangs the wrench. The Broad version increases the initialization range of all three pieces (base, L-shaped hook, and wrench).Coffee and Coffee Broad. The robot must pick up a coffee pod, insert it into a coffee machine, and close the lid. This task has 1 handoff section where the learned agent inserts the pod and closes the lid. The Broad version increases the initialization range of the pod and the coffee machine.Coffee Preparation. This is an extended version of Coffee. The robot must place a mug onto the coffee machine, open the lid, open the drawer with the coffee pod, pick up the pod, insert the pod into the coffee machine, and close the lid. This task has 3 handoff sections where the learned agent (1) places the mug and opens the lid, (2) opens the drawer, and (3) inserts the pod and closes the lid.See Figure 7 for an illustration of all the handoff sections.In Figure 3 , we show the best run out of 5 seeds. Here we provide the mean and standard deviation of the success rates in Table 3 . We observe that although SPIRE still outperforms BC in terms of mean success rate in most of the tasks, our method exhibits unusually high variances in some of the tasks, for example, Coffee, Three Piece, and Tool Hang. In those tasks, one or more runs result in a performance significantly lower than the rest. Specifically,• In Coffee, one run has 40% success rate, while the rest are all 100%;• In Three Piece, one run has 22% success rate, while the rest are at least 98%;• In Tool Hang, one run has 0% success rate and one has 6%, while the rest are at least 82%. Reinforcement learning methods are known to have high variances, especially in sparse reward settings. SPIRE partially alleviates this problem by enforcing the KL penalty for deviating from an anchor policy. However, in practice, such deviation can still happen.Figure 8 compares the training curve of a successful run (with 88% final success rate) and a failed run (with 0% final success rate). The policy in the failed run drastically deviated from the BC policy early on in the training. This is likely related to the unusually large policy gradient loss, which the KL penalty term was unable to match and failed to constrain the policy. In our experiments, such an abrupt decrease in policy gradient loss happens frequently, with varying scales and timing, causing the training results to have high variance. Using an adaptive weight of the KL penalty might be a potential solution, which we wish to investigate in future work.We do not believe 5 seeds are enough to quantitatively reflect the chance of such sudden deviation happening. An alternative solution would be to compare only the results where such deviation did not happen, which is why we chose to report the top-1 performing seed in our main paper.The insert action is deployed using a learned policy π. Its parameters are object o, current object grasp g, current robot configuration q 1 , future configuration q 2 , and learned policy π. The PreInsert(o, g, q 1 , π) precondition models the set of combined object and robot states that initiate the action, thus defining the initial state distribution of the MDP for policy π. The Inserted(o, π) effect models the set of terminal states of the MDP for policy π, namely where the sparse reward r(s) = 1. Unlike move and pick, insert is a stochastic action because end configuration q 2 depends on the execution of policy π. Because of this, we replan after executing each learned action.insert(o : obj, g : grasp, q 1 : conf, q 2 : conf, π : policy) pre: {PreInsert(o, g, q 1 , π), AtGrasp(o, g), AtConf(q 1 )} eff: {Inserted(o, π), HandEmpty(), AtConf( q 2 ), ¬AtGrasp(o, g), ¬AtConf(q 1 )} How do we select the handoff actions?SPIRE offers flexibility to human modelers for deciding which skills should be learned, depending on the precision requirements of the interaction. In our experiments, we used our understanding of the limits of the TAMP system to determine which skills should be learned. The hand-crafted skills, or the classical skills in our work include moving without colliding (transit and transfer motion) and grasping, which are fairly easy to automate. The rest of the skills are learned.In this section, we describe how we bridge the gap between a TAMP planner and RL in full detail. We follow our main text and use S and A to denote the state and action space respectively, with T : S × A → S being the state transition function and S * ⊆ S being the set of goal states.We use TAMP as our default planner in our main text. In this section, we generalize our method to any planner with the following assumptions.A SPIRE-compatible planner P, conditioned on a generated high-level plan, can be defined as a tuple (N, S init , {S i pre }, {S i ef f }) N i=1 , where N is the number of handoff sections; S init ⊆ S is set of the admissible initial states; for handoff section i, S i pre ⊆ S is the set of states where the preconditions for this section are satisfied, while S i ef f ⊆ S is the set where the desired effects are realized. The planner works by iterating through the sections from 1 to N . The planner should satisfy the following properties.Sequence validity. For each section i, if the desired effects of the previous section are satisfied, i.e., the current environment state s ∈ S i-1 ef f (let S 0 ef f = S init for notational convenience), the planner then plans a sequence of actions that always terminates and leads to a new state that satisfies the preconditions of the current section, i.e., s ′ ∈ S i pre . Section validity. For each section i, starting from any state s ∈ S i pre , a state s ′ ∈ S i ef f is reachable within a finite number of steps.Goal validity. Any state that satisfies the effects of the final section should be a goal state, i.e., S N ef f ⊆ S * .SPIRE runs by interleavingly executing the planner P and the agent π, iterating through the N handoff sections. In the i-th section, if the current state s / ∈ S i-1 ef f , then we terminate the process and report failure; otherwise, we execute the planner until the preconditions of this section are met. The planner then hands it off to the agent until the desired effects are met or a specified step limit has been reached. Theorem 1. Let P be a SPIRE-compatible planner, i.e., P satisfies sequence validity, section validity, and goal validity. There exists an agent π * with which SPIRE reaches the goal state deterministically within a finite number of steps if the initial state is admissible by P, i.e., s 0 ∈ S init and with a large enough step limit.Proof. With an induction on the handoff sections, we can conclude that given sequence validity and section validity there exists an agent π * with which SPIRE always reaches a state in S N ef f within a finite number of steps in each section. Given goal validity, the final state is always a goal state.Planner-induced MDPs. In Sec 3.1, we formulate the planner-induced MDPs as a series of N MDPs with shared dynamics and different reward functions and initial state distributions, each corresponding to a planner handoff section. We can now formally define each MDP's reward function:For the i-th MDP, the reward function is r i (s) := 1 {s∈S i ef f } . The support of the initial state distribution satisfies supp(p i 0 ) ⊆ S i pre . Sample collection rate issue. One critical issue we encountered when integrating a TAMP planner into RL training was the dropped rate of sample collection. To give more context, suppose it takes a fixed time of t to sample one frame of environment interaction with an RL agent. Under this assumption, the natural sampling speed with pure RL is 1/t frames per second (FPS). When combined with a TAMP planner, let the planning time before each handoff section be T , and let the RL agent run for at least H steps in each section, the resulting sampling speed becomes at most H/(T + tH) FPS (see Figure 9a ), a relative slowdown of Ht/T . In practice, the TAMP planning time T can be much longer than tH, making this slowdown significant.Initial state distribution issue. We know that the support of the initial state distribution in the i-th section is within S i ef f . However, even if we assume the planner is fixed, the distribution can still change based on the agent behaviors in the previous sections. In the case of training a single agent for all sections, the distribution is even more prone to policy changes as the completion rates of previous sections can affect the appearance rates of later sections.To resolve the two aforementioned issues, we designed the multi-worker scheduling framework that utilizes parallelization, as introduced in Sec 3.5.Solving the sample collection rate issue. See Figure 9b . The multi-worker system runs n TAMP planners in parallel. When a worker finishes planning and reaches the next handoff section, it will notify the scheduler to be available and enter hiatus, until selected by the scheduler for RL interactions. If the system has enough workers, specifically when n ≥ T tH , there always will be a TAMP worker available for RL and reach the optimal sample collection rate of 1/t FPS.Solving the initial state distribution issue. We attempt to alleviate the changing initial state distribution issue by applying a selection strategy to the scheduler. Specifically, the sequential strategy only accepts a section when the success rate of passing all the previous sections reaches a threshold, which indicates that the agent's behavior in previous sections becomes stable. The framework allows for other potentially more sophisticated selection strategies, even though we do not find it necessary in the scope of our work.We provide an additional ablation study on how the sample collection rate boost from the multiworker system benefits the overall performance. We run SPIRE with 16 workers and 1 worker respectively for the same amount of 6-hour wall-clock time, for the first insertion of the Tool Hang task. The result is shown in Table 4 .  


Decision: This paper combines TAMP, behavior cloning, and RL, to built a system that can solve complex long-horizon manipulation problems.

Although on the first sight this work can be seen as a mere combination of TAMP, BC, and RL, this paper is relevant for the community because it bridges the gap between TAMP that often cannot deal with manipulation tasks that require high precision, and RL which struggles with long horizon tasks. This combination and the insights are novel, and hence of value for the community.

The authors provided a strong rebuttal addressing most of the concerns of the reviewers.

Please note that according to one reviewer, the paper still need further major revision for clarity and presentation.

Reviews: review: title: title: Review
paper_summary: paper_summary: A framework based on TAMP utilizing BC and RL.
main_review: main_review: Authors present a framework that is named SPIRE, that relies on TAMP for decomposing the manipulation problem into a sequence of individual skills, similar to HTIL-TAMP. TAMP hands over the control of the robot to human user/RL. Skills are learned with combination of BC and RL. BC is not directly used for warmstarting but for constraining RL policy with divergence loss. Seems that not all the skills are learned but only mentions a hand are in some problem domains, although they are long-horizon.

Strengths:

- Parallelisation, not sure if this is also in the HITL-TAMP, Multiworker.
- Relatively good performance compared to baseline.
- Comparison to two related similar works: HITL-TAMP, TAPM + BC and plan-sequence-learn + TAMP + RL.
- The curriculum approach is interesting.

Weaknesses:

- BC and RL combination is not new, many examples in [1], chapter RL with HiL.
- Many assumptions seem to be hidden, e.g. segmentation, TAMP model, "classical skills", etc. They are also tackled in the questions.

minor:
- In the abstract, Robot Learning is not a technique.
- In the abstract, problem complexity is not proportional to the horizon length, it is exponential.
- Line 27: It is not only teleoperation, it can also be kinesthetic teaching, e.g., [1].
- Citations [7] and [13] are misinterpreted.
    - [13] is not tackling long-horizon problems, it is focused on OOD generalization.
    - [7] is temporal difference learning, and not long-horizon.
- Many citations for some concepts without detailed explanation why [1, 2, 3, 4, 5], it is overwhelming to read and include very general works, more like a survey, but also shallow in that regard. Perhaps in those cases better to refer to some survey if some specific work is not needed.
- Line 85, incomplete sentence.
- Line 93, "learning robot..."
- Line 121, alternating TAMP?
- Line 122, why call hand-off?

[1]Celemin, Carlos, et al. "Interactive imitation learning in robotics: A survey." Foundations and Trends® in Robotics 10.1-2 (2022): 1-197.
questions: questions: Questions:

- Which skills are learned and which are hand-crafted? What are "classical skills"?
- Where does the TAMP model come from?
- How is it decided which tasks are handed over?
- How are symbolic predicates obtained?
- Which planner is used?
- In the pipeline for handing over, is the complete plan from TAMP available?
- TAMP planner seems to use cover part but it is not explained.
- Why is RL failing on those domains?
- How does your method generalize to novel domains?
- Learning curve for RL and SPIRE would be very informative.
limitations: limitations: 2
review_summary: review_summary: Good paper, but nothing really outstanding to be accepted. Seems very incremental with many assumtions that need to be discussed.
title: title: Official Review of Submission628
paper_summary: paper_summary: This paper presents a hybrid learning method aimed at long-horizon manipulation tasks. Within the framework, the TAMP layer executes manipulation for plannable actions. For non-plannable actions, it enters handoff sections. For training, human demonstration data is collected in a human-in-the-loop manner in the handoff sections. Then, policies are initially trained on the demonstration data through behavior cloning and later fine-tuned by RL. During testing, the TAMP planner and these handoff policies are used for manipulation tasks.
main_review: main_review: Strength
- The proposed idea leverages the complementary strengths of TAMP, IL, and RL.
- The quantitative evaluation and the ablation studies well support the design choices of the framework.
- This work demonstrates the successful training of long-horizon tasks with a handful of human demonstrations.
- This paper describes its limitations and the assumptions on the evaluation tasks well.

Weakness
- It would be good to describe the limitation of previous works on TAMP in the Related Works.
- I might have missed the details, but it does not seem to describe how TAMP prompts handoff sections. Additionally, it would be helpful to elaborate on what actions can be considered 'too hard to plan' by the TAMP planner in more precise terms. Are these tasks contact-rich, or do they involve precise manipulation? Why are they difficult to plan by the TAMP planner?
- The paper states that SPIRE does not introduce desirable behavior. However, it seems to be applied only to handoff sections. TAMP may not guarantee desirable actions.
questions: questions: - The proposed method uses TAMP. Therefore, it would be worthwhile to add a vanilla TAMP baseline that does not use the gated RL/BC policy. This would highlight the strength of SPIRE over the TAMP planner alone. If this is not available, is there any way to adapt it for a baseline?
- Is the TAMP used in SPIRE model-based? If so, it might require full geometric information of the environment. However, it seems that the observation space is provided as visual observation from multiple viewpoints. Doesn't this cause partial observability for TAMP?
limitations: limitations: 3
review_summary: review_summary: I think the idea of integrating TAMP, IL, and RL in this paper is interesting. Additionally, the proposed framework is sound, and the results support the strength of the method. However, there are some unclear parts regarding SPIRE’s TAMP layer, as mentioned in my review.
title: title: Major paper organization issues obscure interesting and useful ideas about BC-initialized RL skills in TAMP system
paper_summary: paper_summary: The key idea of this paper is that combining behavior cloning (BC) and RL fine-tuning can lead to completing tasks more efficiently and reliably within a hybrid TAMP system that uses a combination of learned and non-learned skills. Authors state a secondary contribution of suggested techniques for efficient RL finetuning from a BC prior. Experiments show large improvements in success rates (reliability), episode length (efficiency) and demo data-efficiency.
main_review: main_review: **Clarity:**
The main contribution of the paper is hindered by the paper presentation and organization. 
Organization seems easier to fix: Appendix G and H contain information that seems way more important to the main contribution than the RL finetuning techniques described in Section 3.4. Much of the section 3.4 content seems better suited to the appendix and content in Appendix G and H seems more suited to be in the main paper. 
For presentation, I see two consistent issues. The first is excessively vague language for important technical information. The second is presenting less important information with equal or higher emphasis than more important information.

Some key examples from contributions (62-73), which really should be the clearest writing in the paper:

Less important information hiding more important info:  On line 65, the second contribution lists warm-starting with BC (well-explored) and then mentions the multi-worker TAMP framework, which is more novel.

Excessively vague language: the first contribution states that the approach is "synergistic" and then states 3 components that were combined. "Synergy" is a low information-density word. Thus, the  contribution is missing a claim on how it integrates these components in a new way.  An improved first contribution can say something stronger about what the synergy is and what it does. 

Another example of excessively vague knowledge is in lines 132-137 of the problem formulation. The authors provided clarifications in the comments, which are also addressed in the appendix, but the main paper describes it as "TAMP determines."
 


**Originality and significance:** 

~~The first and presumably main contribution seems to be a straightforward and minor modification of [14] by replacing BC policies with BC-initialized RL policies. [15] already does RL policies integrated into a higher-level planning system~~
- The paper addresses the challenge of sampling valid initial states for RL policies that need a large amount of data. Bootstrapping with BC enables safer and more efficient exploration. 

- The ideas presented related to initializing an RL policy with an IL policy is not a “key insight” introduced by this paper, which seems like an overclaim. The KL divergence term and initialization of some weights from a BC policy are known to the authors, which they mentioned in the “RL with experts” section of related work and even show a relationship between their method and [47]. The distinction of prior work being “focused on low-dimensional or single-stage MDP settings” seems insignificant given Huang et al. 2022 and Liu et al. 2022 use image inputs in a driving environment

**Strengths:**
- Multi-worker parallelization makes RL feasible when running the planner to collect in-distribution data.
- RL finetuning does greatly improve the quality of plans executed with HITL-RL
- ~~Paper is generally well organized: information is in the expected sections. It was easy to navigate and find more details.~~
- ~~Contributions are stated clearly~~
- Experiments measure the contributions
-Warmstarting with an IL policy solves the task in a "human"-like way. As authors note, this can be good for safety by generally doing the motion in a "sensible" way
- Contains a robot interacting with the objects in an experimental setting that is practical to exist in the real-world
- As far as insertion and hanging type problem settings go, these are relatively difficult due to the unwieldy shapes of the objects.
- Very nice results in the E2E evaluation
    - contains some (easier) problems where baselines perform the same as the method, which reflects a good-faith implementation of the baselines
    - Significantly improved metrics from BC/RL in several tasks at the bottom of Figure 3.
        - success rate
        - episode length
        - number of needed human demonstrations
- Compared against strong baselines.
- Videos give qualitative information about what the policies look like 
- Limitations section presents assumptions about the system that seem like the most important to me as well. 

**Weaknesses**
- The main weakness is in presentation and organization. More information is in "clarity"
- The first RW paragraph seems focused on HiRL and doesn’t include a discussion comparing with other planning approaches with learned skills.
- Evaluation appears to be entirely in simulation, which makes the safety argument in Figure 4. slightly less compelling. It also shows less potential for dealing with noise/occlusion that happens in the real world but not in simulation.
- The skills used for “handoff” such as insert and hang don’t seem compellling to me as not suitable for motion planning in this domain, especially since these tasks can be quite easy to do kinematically or with visual servoing, especially in manufacturing settings with multiple available camera views.
    - Contact-rich does not imply difficult to specify classically. Contacts can be used to make insertion easier, such as by using some compliance (Mason 1981)
    - If the same exact mechanisms are used as in [14], please say that explicitly. It would also make it clearer what this work does that [14] does not do beyond adding an RL fine-tuning step.
    - What are the "classical" skills supplied? How are they specified? 
- The curriculum learning ablation study seems very specific to that skill and not insightful more broadly.
- Labels in figures do not describe the action represented by the section despite the fact that based on my understanding of [14], those actions are grounded in a specific action with specific preconditions, postconditions, etc… Figure 1 and 7 would be easily improved with such labels. Potentially Figure 4 as well.


**Typos:**
- "supplematary" folder is misspelled in the supplementary material
- a contact-rich skill is called “hand” in the caption of Figure 2 and is probably supposed to say “hang”

**References**

- Huang, Zhiyu, Jingda Wu, and Chen Lv. "Efficient deep reinforcement learning with imitative expert priors for autonomous driving." *IEEE Transactions on Neural Networks and Learning Systems* 34.10 (2022): 7391-7403
- Liu, Haochen, et al. "Improved deep reinforcement learning with expert demonstrations for urban autonomous driving." *2022 IEEE intelligent vehicles symposium (IV)*. IEEE, 2022.
- Mason, Matthew T. "Compliance and force control for computer controlled manipulators." *IEEE Transactions on Systems, Man, and Cybernetics* 11.6 (1981): 418-432.
questions: questions: 1. Please clarify the significance of adding IL initialized / RL fine-tuned policies that is not shown by [14] or [15] What are the challenges and added complexities of finetuning IL policies to be used in a TAMP system that do not apply to BC policies or RL policies in general?
  

2. What TAMP method do you use? At the very least, what is needed from a TAMP system to be compatible with this method? It seems to me implicitly that the TAMP method needs to be [14]. Ex) how are goals specified, how is state information used. A PDDL file in the supplementary would help.

3. Related to above, where do the postconditions come from and how are they used during both IL and RL? Does a demonstration need to satisfy the postcondition? How does the demonstrator know what to do? 

4. If you have the preconditions and postconditions and a simulated environment, why do you need to run the TAMP planner to gather the dataset for learned skills?
limitations: limitations: 3
review_summary: review_summary: Currently, the (main) paper does not clearly communicate the authors ideas about integrating BC-initialized RL policies into their TAMP system. After reading author responses, I can see why the BC/IL addition is a nontrivial and helpful addition to HITL-TAMP. My scores are updated to reflect a better understanding of the technical contribution. I am leaving my recommendation as "Weak Reject" since I think the paper needs major revision beyond the revised manuscript.  I recommend to the authors that they re-organize the paper to highlight the most significant ideas in the main paper.