SPIRE: Synergistic Planning, Imitation, and Reinforcement Learning for Long-Horizon Manipulation
Authors: Zihan Zhou
Animesh Garg
Dieter Fox
Caelan Reed Garrett
Ajay Mandlekar
Venue: CoRL 2024 (2024)
Keywords: Reinforcement Learning
Manipulation Planning
Imitation Learning
TL;DR: Adding RL to Planner guided subtasks in long-horizon BC improves robustness and performance multifold.

Abstract: Robot learning has proven to be a general and effective technique for programming manipulators. Imitation learning is able to teach robots solely from human demonstrations but is bottlenecked by the capabilities of the demonstrations. Reinforcement learning uses exploration to discover better behaviors; however, the space of possible improvements can be too large to start from scratch. And for both techniques, the learning difficulty increases proportional to the length of the manipulation task. Accounting for this, we propose SPIRE, a system that first uses Task and Motion Planning (TAMP) to decompose tasks into smaller learning subproblems and second combines imitation and reinforcement learning to maximize their strengths. We develop novel strategies to train learning agents when deployed in the context of a planning system. We evaluate SPIRE on a suite of long-horizon and contact-rich robot manipulation problems. We find that SPIRE outperforms prior approaches that integrate imitation learning, reinforcement learning, and planning by 35% to 50% in average task performance, is 6 times more data efficient in the number of human demonstrations needed to train proficient agents, and learns to complete tasks nearly twice as efficiently. View https://sites.google.com/view/spire-corl-2024 for more details.

SPIRE: Synergistic Planning, Imitation, and Reinforcement for Long-Horize Manipulation
Zihan Zhou 1,2 Animesh Garg Dieter Fox!, Caelan Garrett Ajay Mandlekar 1 NVIDIA," 2 University of Toronto, Vector Institute Georgia Institute of Technology, equal contribution
Robot learning has proven to be a general and effective approach for programming manipulators, Imitation learning is able teach robots solely from human demonstrations but is bottlenecked by the capabilities of the demonstra- tions. Reinforcement learning uses exploration to discover better behaviors; how- ever, the space of possible improvements can be too large to start from scratch. And for both approaches, the learning difficulty increases exponentially to the length of the manipulation task. Accounting for this, we propose SPIRE, a sys- tem that first uses Task and Motion Planning (TAMP) to decompose tasks into smaller learning subproblems and second combines imitation and reinforcement learning maximize their strengths. We develop novel strategies train learning agents when deployed in the context of planning system. We evaluate SPIRE on a suite of long- mor-zooizon and contact-rich robot manipulation problems. We find that SPIRE outperforms prior approaches that integrate imitation learning, rein- forcement learning, and planning by 35% to 50% in average task performance, 6 times more data efficient in the number of human demonstrations needed to train proficient agents, and learns to complete tasks nearly twice as efficiently. View httss///ites.google.ooom httpe///ooooeeee/s/reoooo-----00 for more details.
Abstract:
Keywords: Reinforcement Learning, Manipulation Planning, Imitation Learning
Introduction
Reinforcement Learning (RL) a powerful tool that has been widely deployed to solve robot ma- nipulation tasks [1, 2. 3, 4]. The RL trial-and-error process allows an agent automatically discover solutions a task and improve its behavior over time. However, in practice, it often relies care- ful reward engineering to guide the exploration process [5, 6]. The exploration burden and reward engineering problem is even more challenging to overcome for long- mogghorizoon tasks, where an agent must complete several subtasks in sequence in order to solve the task [7].
Imitation Learning (IL) from human demonstrations [8, is popular alternative to reinforcement learning. Here, humans teleoperate robot arms collect task demonstrations. Then, policies trained using the data. This alleviates the burden of reward engineering, since correct behaviors directly specified through demonstrations. This paradigm has recently been scaled up by collecting large datasets with teams of human operators and robots and shown to be effective for different real-world manipulation tasks [10, 11, 12]. While these agents can be effective, they typically are imperfect, with respect to both success rates and control cost, and not robust to different deployment conditions, especially when it comes to long- morizoo tasks [13].
One way to integrate the benefits of both IL and RL is to first train agent with IL and then finetune it with RI. This can help improve the II. agent and make it robust through trial-and-error, while also alleviating the need for reward engineering due to the presence of the demonstrations. Several works have used this paradigm successfully, but long-horizon manipulation still remains challenging due the burden exploration and long-term credit assignment [7].
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Task
TAMP
Coffee
Preparation
1: mug
2: Open lid
3: Open drawer
4: Insert pod
Section
Section 2
Telc-op BC
RL Policy
KL
BC Policy
RL fine-tune
Section
Section 4
Figure 1: SPIRE Overview. (Left) SPIRE first attempts to salve the task with TAMP system. When the TAMP planner encounters action deemed too hard to plan, it then enters the handoff' section and delegates the action a human teleoperator to manually complete
record the trajectories from the human operators to build a demonstration dataset and train an IL policy with it. Finally, we train an RL policy to fine-tune the IL policy via warmstarting and deviation constraining, (Right) The four handoff sections in Coffee Preparation.
One effective approach for long-horizon manipulation to leverage hybrid control paradigm, where the agent is only responsible for local manipulation skills, instead of the full task [14, 15]. An example is the HITL-TAMP system [14], where an agent is trained with IL small contact- rich segments of each tasks, and the rest of the task is performed using Task and Motion Planning (TAMP) [16]. Another related approach is PSL [15], which learns an agent using RL instead of IL on small segments, and uses motion planning to sequence the learned skills together. These approaches are effective for challenging long-horizon manipulation tasks, but they still often do not train perfect policies, suffering from some the pitfalls of IL and RL. In this paper, we take inspiration from these approaches and create hybrid control learning framework that allows for efficient imitation learning and RL-based finetuning of agents to address morizon manipulation tasks.
We introduce Synergistic Planning Imitation and REinforcement (SPIRE), a system for solving challenging long-horizon manipulation tasks through efficient imitation learning and RL-based fine- tuning. SPIRE decomposes each task into local manipulation segments that are learned with a policy and segments handled by a TAMP system. The manipulation segments are first trained via imitation learning and then finetuned with reinforcement learning. Our approach ing manipulation tasks reaches an average success rate of 87.8%, vastly outperforms TAMP-gates IL [14] (52.9%) and RL [15] (37.6%). In the subset of tasks where SPIRE and IL both reach a high success rate, our method only uses 59% of the steps required by IL to complete the task. In Tool Hang, SPIRE fine-tunes an II. policy with only 10% success rate to 94%. We perform a thorough analysis of our method and also show that in many cases, a handful of demonstrations suffice for learning challenging tasks. Compared with IL, SPIRE improves the overall demo efficiency by 5.8 times in the evaluated subset of tasks.
9 challeng-
Our contributions are as follows:
system that synergistically integrates the strengths of behavior cloning, reinforcement learning, and manipulation planning. SPIRE first learns a TAMP-gater policy with BC and then improves it with RL.
We propose SPIRE, a hybrid
We introduce key insights to enable RL-based finetuning with sparse rewards in this regime, in- cluding mechanism warmstart the RL process using the trained BC policy, way constrain exploration to be close to the BC agent behavior, and multi-worke TAMP framework to optimize the throughput of SPIRE's RL process.
We evaluate SPIRE on a suite of long-horizon contact-rich tasks and find that it outperforms prior hybrid learning-planning approaches success rate averaged across tasks (87.8%, compared to 52.9% and 37.6%), execution efficiency (episodes are only 59% the length of the BC agent), and human demonstration efficiency (6 times less data required than BC train similar agents).
Related Work
Hierarchical approaches for long-horizon tasks. Hierarchical approaches decompose the chal- lenging long-horizon tasks into easier-to-solve subtasks. RL based methods explore the division of

---

sub-tasks with reusable skills [17, 18, 19. 20, 21]. [22, 23, 24, 25, 26] build hierarchical RL solutions with subpolicies and metacontrollers. Our work instead leverages a planner that provides guidance on which policies to learn as well as initial and terminal state distributions of tasks, compared to bottom-up HiRL methods, which tend to be data inefficient. Notably this top-down breakdown may also be achieved with a Language Model which can provide plan composed of steps and sub-goal targets [27, 28, 29, 30, 31, 15].
Robot manipulation with demonstrations. Behavior cloning (BC) [32] learns policy by directly mapping observations to actions and is typically trained end-to-end using pre-collected pairs of ob- servation and behavior data. While this is seemingly a supervised learning problem, the context of robotics adds challenges. BC datasets tend to contain data sampled from multimodal distributions, due intra-expert variations. Recent work address this problem with implicit models including those derived from energy-based models [33, 34], diffusion models [35, 36, 37, 38, 39], and trans- former models [40, 41, 42]. Another challenge is the correlation in sequential data, which can lead to policies which are susceptible to temporally correlated confounders [43]. Recently several works have set out to handle this by predicting action chunks. For example, the Action Chunking Transformer (ACT) line work [44, 45] and diffusion policy [35]. While BC-based methods com- bined with high-capacity models enable complex robotics tasks from demonstrations, problems in robustness and long-horizon generalization remain.
Experts and their demonstrations can be used in combination with RL in multiple ways, including acting as task specifications, exploration guidance, and data augmentation. Inverse RL [46, 47, 48] learns a reward model for RL. from demonstrations; [49, 50, 51] explore using demonstrations to bootstrap the reinforcement learning process; [52] uses state matching for reward computation in RL. [53] shares similar setup with ours, where they also warmstart RL with a BC policy and use masked BC loss the constrain the RL policy from deviating. [54, 55] propose fine-tune semi- expert initial policy by training a residual policy on top of it with RL. Different from the mentioned works, we focus on multi-stage robotic manipulation tasks with high-dimensional input spaces.
Learning for Task and Motion Planning. Task and Motion Planning (TAMP) is a search and planning- based approach for sequential manipulation tasks [16, 56, 57, 58). However, pure TAMP- based methods suffer from reliance on accurate environment modeling and prior knowledge of the skills, making those methods less suitable for tasks involving complex contact-rich skills.
3 Method
Our approach Synergistic Planning Imitation and REinforcement (SPIRE) learns and deploys closed-loop visuomotor skills within TAMP system (see Fig. 1). First frame problem as policy learning problem across a sequence of Markov Decision Processes (Sec. 3. 3.1). )). Next, we describe our approach for incorporating both classical and learned robot skills into TAMP (Sec. 3.2) to enable TAMP-gated learning. Next, we describe how we train an initial agent with TAMP-gated Behavioral Cloning (BC) (Sec. 3.3). We then propose an RL-based finetuning algorithm to improve the BC agent with RL (Sec. 3.4). Finally, we introduce a parallelized training scheduler that is able intelligently manage dependencies among stages when conducting RL in our setting (Sec. 3.5).
3.1 Problem Formulation
In our setup, each robot manipulation task can be decomposed into a series of alternating TAMP sections and handoff sections, where TAMP delegates control to a trained agent т. These sections are TAMP-gated [1 [14], as they are chosen at the discretion of the TAMP system, and typically involve skills that are difficult to automate with model-based planning. We wish to train an agent æ to complete these handoff sections efficiently and reliably. We model our TAMP-gated policy learning problem as a series of Markov Decision Processes (MDPs), M = (S, A, (r'), {pa), 7) where N is the number of MDPs (each corresponds to TAMP handoff section), S and A are the state and action space. T is the transition dynamics, rs) and p are the i-th reward function and initial state distribution, and y is the discount factor. The start and end of each handoff section is chosen by TAMP, consequently, TAMP determines the initial state distribution pó for each handoff
3

---

section, and the reward function (s), which is a sparse 0-1 success reward based on the successful completion of the section. Our goal is to train a stochastic policy я : S - Aa that maximizes the expected return J(T) == B(i E== 7"ri]. We next describe the TAMP system.
3.2 TAMP with Learned Skills
Task and Motion Planning (TAMP) [16] is a model-based approach for synthesizing long-horizon robot behavior. TAMP integrates discrete (symbolic) planning with continuous (motion) planning to plan hybrid discrete-contimuuus manipulation actions. Essential TAMP is a model of the ac- tions that a planner apply and how these actions modify the current state. From such model, TAMP solvers can search over the space of plans to find a sequence of actions and their associated parameters that satisfies the task.
In SPIRE, we seek to implicitly learn a select set of TAMP actions that are impractical to manually model and then combine them with traditional actions through planning. In essence, our strategy is to learn policies that control the system from TAMP precondition states to postcondition states (typically described by effects). We adopt the modeling strategy introduced by Mandlekar et al. [14] and deploy PDDLStream [58] to solve each TAMP problem. See Appendix G for summary of our planning model. Fig. 2 visualizes an interleaved execution of TAMP trajectory control and RL-learned policy control in the Tool Hang domain. Here, the planning model explicitly models the pick as well as intermediate move actions but defers the insert and hang actions to the RL. agent.
Algorithm describes the SPIRE policy at test time. It begins by observing the state. If the state sat- isfies the task's goal conditions, SPIRE terminates successfully. Otherwise, SPIRE invokes TAMP to plan sequence of traditional and learned actions. SPIRE executes the trajectory associated with each traditional action until it reaches the first learned action. At that time, SPIRE executes the closed-loop policy associated with the learned action until achieves its subgoal condition. To account for the stochastic outcome of the policy, SPIRE replans and repeats this process.
3.3 Mate- Imitation Learning
Algorithm SPIRE
TAMP-Gated Data Collection. We collect an initial dataset of human demonstrations through TAMP-gated human tele- operation, where the human operator collects demonstrations for handoff sections when prompted by TAMP, to form the demonstration dataset D {[(s), . a, A and H. is the horizon, and ga is the handoff section of the i-th trajectory. To improve the data collection efficiency,
1: procedure SPIRE(G) while True do OBSERVE()
if E then return True ă < PLAN-TAMP(s. G)
6:
ge.).
where s S,
7:
8:
if a. "RI." then
9:
a.polyy EXBCUTE-POLICY(S)) break else
10:
we replicate the task queuing system from [14].
12:
TAMP-Gated Behavioral Cloning. Given the dataset
14:
EXECUTE TRAJECTORY(T)
a. trojjeetooyy
D. we train Behavioral Cloning (BC) policy parame-
minimize the negative log-likelihood loss over the demonstration dataset: o== argming [(s.a)=D[- log (a(as)) The trained BC agent тÓ may have substantial room for improvement, depending on the complexity of the task, and the number of demonstrations available for training. We next describe our RL-based finetuning procedure (Sec. 3.4) that allows this agent to be improved through reinforcement learning.
terized by
3.4 RL Finetuning
Given trained BC agent Tg. wish train an RL agent Ta improve performance further. To avoid reward engineering, we only assume access to sparse 0-1 completion rewards for each hand- off section provided by TAMP (Sec. 3.1). However, exploration in sparse-reward settings has been shown to be challenging [59, 60, 61, 62], especially in continuous state and action spaces. Fortu- nately, we can use the BC policy trained in the previous section as a reference point for exploration we want to restrict the behavior of the RL policy to be in a neighborhood of the BC policy. This is achieved by a) warmstarting the RL policy optimization using the BC policy, and b) enforeing a constraint the deviation between the RL policy and the BC policy.

---

pick
insert()
pick
hang ()
RL Pollcy
RL Policy
Su
EED
50
(F(
pre
eff
pre
eff
Figure 2: SPIRE execution. SPIRE computes a TAMP plan but defers execution certain contact-rich skills, such as insert and hang. to learned agents we call these handoff sections. The preconditions of each handoff section define the initial state distribution of the agent, and the postconditions of each action correspond the termination states of the corresponding MDP for the handoff section.
Warmstarting RL optimization with BC. We tested two ways to warmstart the RL agent. Ini- tialization. One method is to initialize the weights of the RL agent with those of the trained BC agent, 8 < Φ" where RL objectives. Despite being easy implement, this can be less flexible since it requires the agent structure of the RL and BC policies to match. Furthermore, researchers have found that retraining neural networks with different objectives can cause the network to lose plasticity [63], which can make the policy harder to optimize because of the objective shift from BC to RL. Residual Policy. An alternative way is to fix the BC policy as a reference policy and train a residual policy on top of it. Let the residual policy be TH (s). The residual policy shares the same action space as the normal policy but is initialized to close to zero. The final action is defined action a Tg- (s) and the residual action at Th (s). practice, we only add the mean of the reference policy the residual action instead of sampling the reference action.
arg ming (Bc(Ø), and subsequently finetune the weights with online
summation of the reference
Constraining Deviation between BC and RL agents. The sparsity of reward signals produces high-variance optimization objectives, which can lead the RL policy to quickly drift away from BC and lose the exploration bonus from warmstarting. Therefore, it is critical to constrain the policy output to be close to the BC agent throughout the training process. We achieve this by imposing KL-divergence penalty. We conclude our RL optimization objective as follows: JFT(0) == J(ro) aD (Tt) (T>)) where (kl() (s)))>> log p(als) and a is the weight for the penalty term. q(a(s)]
3.5
Multi-Worker Scheduling Framework
Making our TAMP-gated framework compatible with modern reinforcement learning procedures requires addressing several challenges. First, TAMP can take dozens of seconds for a single rollout, which severely lowers the throughput of RL exploration. Second, the TAMP pipeline executes each section sequentially, which means that later handoff segments can only be sampled when previous handoff segments are completed successfully. This leads an imbalance of episodes for the differ- ent handoff segments and is potentially problematic for the RL agent. We propose multi-worker TAMP scheduling framework to integrate TAMP into RL fine-tuning. The framework consists of three components group of TAMP workers that run planning in parallel, a status pool that stores the progress of the workers, and scheduler that distributes tasks to the workers and balances the initial states. We further describe how the framework allows for curriculum learning, and how the framework accelerates learning efficiency for RL training. See Appendix H for more details.
TAMP workers. Each TAMP worker has an environment instance and repeatedly runs
Algorithm Scheduler
a TAMP planner. Upon reset, the TAMP worker initiates TAMP until a handoff sec- tion has been reached. It then sends a pair (#worker, #section) representing its ID and which handoff section has entered FIFO status queue, indicating that it is ready to take RL agent actions. The worker then enters an idle state until it receives command from
1:
procedure SCHEDULER(WORKEN STATUSQUEUE, POLICY, STRATRGYY)
2:
while Truc do
3:
i. < STNTUSQUEUE,
4.
P.STRATEEY..ccege
5:
while not WORKERS (il.sssiidddddd do
Soia WORKERS (errkERs(s].oseeeee observu)))
7:
+- POLICY act(
8:
WORKERS(E)
9: else
10:
WORKERS (OOKKEs(()).rsst))))
5

---

Square
Episode Duration:
BC[14]
RL [15]
Ours
Square B.
Square
18.1
11.6
Coffee
Coffee B.
Square Broad
24.5
8.4
13.6
CoffeePrep
Coffee
63.1
15.0
38.4
TPiece
Coffee Broad
80.6
25.7
61.3
TPiece B.
Coffee Preparation
193.3
168.5
ToolHang
Three Piece
58.7
34.0
THng B.
Three Piece Broad
38.1
BC
20
40
60
80
100
Tool Hang
61.7
succ. rate (%)
Tool Hang Broad
130.5
109.8
Ours
Figure 3: Full evaluation. Comparing the success rates (left) and the average duration (right) of successful rollouts of HITL-TAMP-BC (BC), TAMP-gated Plan-Seq-Learn (RL), and SPIRE (Ours) across all 9 tasks. Each datapoint chosen from the best run out of seeds and averaged from 50 rollouts. SPIRE improves the BC policy in terms of both success rate and average duration in all 9 tasks and reaches 80% success rate in 8. RL has an advantage in average duration in the easier set of tasks but fails to learn anything in the rest.
the scheduler. Depending the command, the worker either resets itself or starts interacting with the environment by exchanging actions and states with the scheduler. If the current section is solved, the worker sends success signal to the scheduler and runs TAMP until the next handoff section.
Scheduler. (Algorithm 2) The scheduler a centralized component that manages the TAMP work- ers. It also provides an environment abstraction the angle-threaded RL process. The scheduler is configured with sampling strategy. Upon initialization, it first pops an item from the status queue. According to the sampling strategy, the scheduler either rejects this section, in which case it sends resetting signal to the corresponding worker; starts new episode and interacts with the worker.
Curriculum Learning. The behavior of the scheduler depends on a sampling strategy, allowing it function as curriculum for the RL agent. We consider two strategies: permissive is the default strategy that allows all sections through, while sequential only accepts section when the success rate of passing all the previous sections reaches threshold. sequential allows controlling the initial state distribution during the early stages of training, ensure the RL agent achieves proficience in each section sequentially before continuing onto the next section.
Remarks on Efficiency. Suppose a TAMP planning process takes at most seconds over the episode; each environment interaction step, counting communication latency, takes at least sec- onds; and each handoff section is least H/ steps. If the number of TAMP workers > i the proposed multi-worker TAMP scheduling framework reaches throughput of at least 1/t frames per second. In comparison, the single- worker counterpart has worst-case throughput of T+H frames second. Suppose that the planning process is slower than the handoff sections by factor k (e.g. T = k LH), then our framework is faster than the worker alternative by a factor of k + 1.
T
H
4 Experiments
Tasks. For evaluation, we follow [14] and choose a set of long-horizon manipulation tasks, namely Square, Coffee, Three Piece, and Tool Hang. We also include the broad variants of those tasks, where we use broad object initialization region, and Coffee Prepuration, which has the longest horizon with four handoff sections. See Appendix for more details.
Environment Details. Observation space. For most tasks, use single 84 x 84 RGB image from the wrist-view camera. For Tool Hang. we use the front-view camera instead since the wrist- view is mostly occluded. For Tool Hang Broad and Coffee Preparation, we use both wrist-view and front-view cameras, as well as proprioception state (end-effector pose and gripper finger width). Action space. Actions are 7-dimensional (3-dim delta end-effector position, 3-dim delta end-effector rotation, I-dim gripper actuation). Horizon. Each handoff section is limited to 100 steps (5 seconds with 20Hz control frequency) for all tasks, except for Tool Hang Broad, where the limit is 200 steps.
Baselines. We compare our method with two baselines: HITL-TAMP-BC (BC), which is adapted from [14] to match our network structure; and TAMP- -ated Seq- (RL), which adapted

---

(a) Naive RI. agent
(b) SPIRE agent
Figure 4: Qualitative comparison. Rollouts of vanilla RL vs our method. The first agent attempts to close the lid by knocking the coffee machine, while our agent follows the demonstrations and closes the lid with fingers.
from [15] by replacing the LLMM-based planning system with our TAMP system for fair comparison. We collected 200 human demonstrations for each task train the behavior cloning policy. For RL, use DrQ-v2 v2 [64]
the base algorithm. See Appendix B for more details.
Evaluation. We evaluate each trained agent for 50 rollouts and report the success rate and average completion steps in the successful rollouts. We train seeds for each algorithm and report the best- performing agent (success rate-wise, tie-breaking with average steps) unless otherwise specified.
4.1 Results
SPIRE outperforms both TAMP-gated BC and RL. We compare our method with the TAMP- gated BC [14] and RL [15] baselines across all 9 tasks (see Fig, 3). SPIRE reaches 80% success rate in 8 out of 9 tasks, while BC and RL only reach 80% in tasks each. In Tool Hang, our method reaches 94% success rate despite the BC counterpart only having 10%, which is over 9-times improvement. Remarkably, this low- oow-performing BC agent is enough to help address the exploration burden (unlike RL, 0% success) and train a near-perfect agent. Across all 9 tasks, SPIRE averages 87.8% success rate, while BC and RL only average 52.9% and 37.6% respectively.
SPIRE produces more efficient agents than BC through RL fine-tuning, SPIRE agents have lower average completion times than their BC and RL counterparts (Fig. 3, right). Even in tasks such as Square, Square Broad, Coffee, Three Piece, where BC policies already have high success rates, our method improves the efficiency by only using an average of 59% completion time.
SPIRE's use of the BC agent helps address the RL exploration burden on challenging long- horizon tasks. Exploration in RL with sparse rewards is extremely challenging, especially for robot manipulation tasks for their continuous and high-dimensional observation and action space. Our method solves the initial exploration problem by anchoring policy learning around the BC agent. As shown in Figure 3, RL policies without utilizing BC only reach nonzero success rates Square, Coffee and their Broad variants, all of which have only one handoff section and relatively shorter horizons. Even in Coffee Broad, RL encounters exploration difficulties due to the broader object distribution, resulting in only partially solving the task.
Qualitatively, SPIRE can improve agent behavior without introducing undesirable behavior, unlike RL. Safety awareness has always been a critical matter in robotics learning. Safety con- straints can be hard to define with numerical values, which adds to the challenges of realizing safety in RL. We notice that in Coffee, RL policy has a much shorter completion time than our method. This is at the cost of ignoring safety concerns. We compare two rollouts of RL and our method in Figure 4. The RI.-trained policy attempts to close the lid by knocking the coffee machine with the arm, which can potentially damage the robot and the coffee machine and even cause danger to humans; while our method preserves safety awareness by following the demonstration's 's practice of closing the lid with its fingers.
SPIRE can train proficient agents using just handful of human demonstrations. BC methods can require several human demonstrations to train proficient agents, which can be major drawback due to the cost of collecting this data [9]. We reduce the number of human demonstrations used by SPIRE to 10 and 50 (instead of 200 as in Fig. 3), and we plot the minimum of demonstrations needed to reach at least 80% success rate in Fig. 5. As the plot shows, SPIRE can successfully fine-tune BC policy trained with as few as 10 demos in evaluated tasks except for Tool Hung
7

---

mininun deros
286
success rate (%)
100
75
50
10
Square
Square
Broad
Coffee
TPiece-
TPiece
Broad
ToolHang
Coffee
Ours
permissive sequential
Figure 5: Demo efficiency and sampling strategy ablation. (Left) Minimum number of demos needed to reach at least 80% success rate. (Right) Success rates across 5 seeds in Tool Hang, comparing permissive and sequential strategies. sequential has a lower variance but permissive has the better top-1 policy.
and Coffee Broad, for which 50 demos are enough. In the 7 tasks, our method needs 150 demos in total, while BC needs more than 870, a 5.8x improvement in efficiency.
4.2 Ablation Study
We conduct two ablative studies to investigate (1) the value of the KL-divergence penalty and (2) the value of curriculum learning, governed by the two scheduler sampling strategies permissive and sequential. In this section, we compare the performance distribution of the 5 runs instead of only the top-1 run for a more comprehensive evaluation.
Value of divergence penalty. We ablate the divergence penalty two representative tasks, Three Piece & Tool Hang (Table 6) and observe drastic performance drop (84% to 17.6%, 74% to 0%).
The sparsity in rewards leads to high-variance opti- mization objectives for RI. As result, even when warmstarted with BC, the RL policy can quickly de- viate from it, especially when the chance of reach- ing the reward signal is low. Therefore, constraining the policy close to BC throughout the training is crit- ical. We select two representative tasks, Three Piece and Tool Hang for this ablation. The result shown in Figure 6. Without the divergence penalty, the RL policies deviated immediately and never returned.
Task
(%)
w/o (%)
Three Piece
84.0 (34.7)
17.6 (39.4)
Tool Hang
74.4 (11.8)
0.0 (0.0)
Figure 6: KL-divergence penalty ablation. Mean and standard deviation (in parenthesis) of across the 5 seeds in Three Piece and Tool Hang. with and without Kl.-divereence penalty. In both tasks, the divergence penalty im- proves the performance by large margin.
Value of curriculum learning. We compare the two sampling strategies in Tool Hang task. The re- sult is shown in Figure 5. sequential strategy shows much smaller variance, while permissive produces the better top-1 seed performance. The main difference between the two strategies is how the second section states emerge during training. For permissive. the second section states emerge gradually as the success rate of passing the first section gets higher, resulting in more gentle dis- tributional shift that leads a higher overall success rate; for sequential, the shift is more abrupt, but it has fewer distraction states in the early stage, resulting in a more stable training process.
Conclusion
We presented SPIRE, an integrated approach for deploying RI, BC, and planning harmoniously. We showed how BC be used not only warm-start RL but also guide the RL process via focused exploration. We introduced scheduling mechanism improve RL data throughput and in- crease learning efficiency. Finally, we evaluated SPIRE in simulation against recent hy brid lcarning- planning baselines and found that SPIRE results in more successful and efficient policies.
Limitations. We focus on tasks that center around object-centric manipulation of rigid objects in table-top environments. We assume that human teleoperator can demonstrate the learned skills to warmstart RL. The TAMP component assumes that the state is observable and comprised of rigid objects, possibly connected with articulation. To simplify RL training, we only considered Markovian policies; however, using neural network architectures with history, such as RNNs, may boost performance [9]. Finetuning BC policy with RL requires the simulation to be efficient.

---

Acknowledgments
The authors thank Vector Institute for computing.
References
[1] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Jang, D. Quillen, E. Holly,
M. Kalakrishnan, V. Vanhoucke, al. Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on robot learning. pages 651-673. PMLR, 2018.
[2] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.088: 12, 2021.
[3] B. Tang, M. A. Lin, I. Akinola, A. Handa, G. S. Sukhatme, F. Ramos, D. Fox, and Y. Narang. Industreal: Transferring contact-rich assembly tasks from simulation to reality. arXiv preprint arXiv:2305.17110, 2023.
[4] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5977-5984. IEEE, 2023.
[5] D. Silver, S. Singh, D. Precup, and R. S. Sutton. Reward is enough. Artificial Intelligence, 299: 103535, 2021.
[6] A. Y. Ng. S. Russell, al. Algorithms for inverse reinforcement learning. In leml, volume 1, page 2000.
[7] R. S. Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3:9-44, 1988.
[8] Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. arXiv preprint arXiv:1710.04615 2017.
[9] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
[10] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint 17200:22/..06817, 2022.
[11] Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and
S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXhv:2109.13396, 2021.
[12] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learn- ing. pages 991-1002. PMLR, 2022.
[13] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[14] A. Mandlekar, C. Garrett, D. Xu, and D. Fox. Human in -othhe-oopp task and motion planning for imitation learning, In 7th Annual Conference Robot Learning, 2023.

---

[15] M. Dalal, T. Chiruvolu, D. Chaplot, and R. Salakhutdinov. Plan-seq-learm: Language model guided rl for solving long horizon robotics tasks. 2024.
[16] C. R. Garrett, R. Chitmis, R. Holladay, B. Kim, T. Silver, P. Kaelbling, and T. Lozano- Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-293, 2021.
[17] Dayan and G. E. Hinton. Feudal reinforcement learning. In NIPS, 1992.
[18) R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell., 1 112:181-211 211- 1999.
[19] P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. AAAI, 2017.
[20] A. S. Vezhnevets, S. Osindero, T. Schaul, N. M. O. Heess, M. Jaderberg, D. Silver, and
K. Kavukeuoglu. Feudal networks for hierarchical reinforcement learning. ICML, 2017.
[21] O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. In NeurlPS, 2018.
[22] T. D. Kulkarni, K. Narasimhan, A. Saeedi, and J. B. Tenenbaum. Hierarchical deep reinforce- ment learning: Integrating temporal abstraction and intrinsic motivation. In NIPS, 2016.
[23] D. Lyu, F. Yang, B. Liu, and S. M. Gustafson. Sdrl: Interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In AAAI, 2019.
[24] Rafati and D. C. Noelle. Learning representations in model-free hierarchical reinforcement learning, AAAI, 2019.
[25] S. Sohn, H. Woo, J. Choi, and H. Lee. Meta reinforcement leamning with autonomous inference of subtask dependencies. ICLR, 2020.
[26] R. Costales, S. Iqbal, and F. Sha. Possibility before utility: Learning and using hicrarchical affordances. ICLR, 2022.
[27] D. Driess, F. Xia, M. S. Sajjadi, C, Lynch, A. Chowdhery, B. Ichter, A. Wahid, Tompson, Q. Vuong, T. Yu, al. Palm-e: embodied multimodal language model. arXiv preprint arXiv:2303.03377. 2023.
[28] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox. J. Thomason, and A. Garg. Progprompt: program generation for situated robot task planning using large language models. Autonomous Robots, 47(8) 1012, 2023.
[29] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishman, K. Hausman, A. Herzog, D. Ho. J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang. R. J. Ruano, Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang. K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu. S. Xu, M. Yan, and A. Zeng. Do as can and not as say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.0169), 2022.
[30] N. Yoshikawa, M. Skreta, K. Darvish, S. Arellano-Rubach, Z. Ji, L. Bjørn Kristensen, A. Z. Li, Y. Zhao, H. Xu, A. Kuramshin, et al. Large language models for chemistry robotics. Autonomous Robots, 1058:1055- 2023.
[31] M. Skreta, Z. Zhou, L. Yuan, K. Darvish, A. Aspuru-Guzik, and A. Garg. RePLan: Robotic Replanning with Perception and Language Models. arXiv e-prints, art. arXiv:2401.0411 Jan. 2024. doi: 10..48550//xxx.2441..00
10

---

[32] D. A. Pomerleau. Alvinm: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
[33] Florence, C. Lynch, A. Zeng, O. A. Ramirez, Wahid, L., Downs, Wong, J. Lee, I. Mor- datch, and Tompson. Implicit behavioral cloning. In Conference Robot Learning, pages 158-168. PMLR, 2022.
[34] D. Jarrett, I. Bica, and M. van der Schaar. Strictly batch imitation learning by energy-based distribution matching. Advances in Neural Information Processing Systems, 33:7354-7365, 2020.
[35] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[36] Y. Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy. arXiv preprint arXiv:2403.03954 2024.
[37] L.. Chen, S. Bahl, and D. Pathak. Playfusion: Skill acquisition via diffusion from language- annotated play. In Conference Robot Learning, pages 2012-2029. PMLR, 2023.
[38] Z. Xian, N. Gkanatsios, T. Gervet, T.-W. Ke, and K. Fragkiadaki. Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In 7th Annual Conference on Robot Learning, 2023.
[39] M. Janner, Y. Y.Du, J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint 1rXiv:2205.09991 2022.
[40] N. M. Shafiullah, Z. Cui, A. A. Altanzaya, and L.. Pinto. Behavior transformers: Cloning modes with one stone. Advances in neural information processing systems, 35:22955 --9968,, 2022.
[41] Z. J. Cui, Y. Wang, N. M. M. Shafiullah, and L. Pinto. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210. 10047, 2022.
[42] S. Lee, Y. Wang, H. Etukuru, H.J. Kim, N. M. M. Shafiullah, and L.. Pinto. Behavior generation with latent actions. arXiv preprint arXiv:2403.0 03181, 2024.
[43] G. Swamy, S. Choudhury, J. A. Bagnell, and Z. S. Wu. Causal imitation learning under tem- porally correlated noise, 2022.
[44] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware, 2023.
[45] Z. Fu, Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02111 2024.
[46] J. Ho and S. Ermon. Generative adversarial imitation learning. In Neural Information Process- ing Systems, 2016.
[47] Ho, K. Gupta, and S. Ermon. Model-free imitation learning with policy optimization. ArXiv, hbss/605508878., 2016.
[48] C. Finn, P. F. Christiano, P. Abbeel, and S. Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. ArXiv, abs/161 1.03852, 2016.
[49] T. Hester, M. Vecerík, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, A. Sendonaris, G. Dulac- Arnold, I. Osband, J. P. Agapiou, J. Z. Leibo, and A. Gruslys. Learning from demonstrations for real world reinforcement learning. ArXiv, abs/1774.03732, 2017.
11

---

[50] M. Vecerik, T. Hester, J. Scholz, F. Wang. O. Pietquin, B. Piot, N. M. O. Heess, T. Rothörl, Lampe, and M. A. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. ArXiv, abs/1707.088" 17, 2017.
[51] V. G. Goecks, G. M. Gremillion, J. Lawhern, J. Valasek, and N. R. Waytowich. Integrating behavior cloning and reinforcement learning for improved performance in dense and sparse reward environments. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 465-473, 2020. 2020..
[52] S. Haldar, J. Pari, A. Rai, and L. Pinto. Teach robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint arXiv:2303.01-99) 2023.
[53] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming explo- ration in reinforcement learning with demonstrations, 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 2292-6299 2017.
[54] Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine. Residual reinforcement learning for robot control. 2019 International Conference on Robotics and Automation (ICRA), pages 6023-6029, 2018.
[55] T. Silver, K. R. Allen, J. B. Tenenbaum, and I.. P. Kaelbling. Residual policy learning. ArXiv, abs/1812.002988, 2018.
[56] L. P. Kaclbling and T. Lozano-Pérez. Hierarchical task and motion planning in the now. In ICRA, 2011.
[57] M. Toussaint, K. Allen, K. Smith, and J. Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. 2018.
[58] C. R. Garrett, T. Lozano-Pérez, and L. P. Kaelbling. PDDLStream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. Proceedings of the ternational Conference on Automated Planning and Scheduling. volume 30, pages 440-448, 2020.
[59] B. C. Stadie, S. Levine, and P. Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. ArXiv, abs/1 01777008144 14, 2015.
[60] H. Tang, R. Houthooft, D. Foote, A. Stooke, X. Chen, Y. Duan, J. Schulman, F. D. Turck, and P. Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. NIPS, 2017.
[61] A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kapturowski, O. Tieleman, M. Arjovsky, A. Pritzel, A. Bolt, and C. Blundell. Never give up: Learning directed exploration strategies. ICLR, 2020.
[62] A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and Clune. First return then explore. Nature, 590 7847:580-586, 2021.
[63] Z. Abbas, R. Zhao, J. Modayil, A. White, and M. Machado. Loss of plasticity in continual deep reinforcement learning. ArXiv, Abs/2303300500) 2023.
[64] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2 2021.
12

---

A Overview
The Appendix contains the following content.
Policy Learning Details (Appendix B)): details on hyperparameters used
Ablation: SPIRE without TAMP (Appendix C): ablation study on the effect of removing
TAMP-gating and directly running BC and RL fine-tuning
Comparison to Additional Methods (Appendix D): comparison to other RL methods that leverage demonstrations
Tasks (Appendix E): details on tasks used to evaluate SPIRE
Variance Across Seeds (Appendix F): discussion on the variance of results across different seeds and how results are presented
TAMP Formulation G): details on the TAMP planner
Bridging TAMP Planner and RL (Appendix H): details how we integrate the TAMP planner with RL
Ablation: SPIRE without Multi- (Appendix I): ablation study the effect of using multiple parallelized TAMP workers
Additional Experiment Results (Appendix J): additional experiment results, including
RL learning curves
13

---

B Policy Learning Details
Table 1: DrQ-v2 hyperparameters.
Network structure
CNN
Learning rate
le-4
Discount
0.99
Batch size
256
n--tep returns
3
Action repeat
1
Seed frames
4000
Feature dim
50
Hidden dim
1024
Optimizer
Adam
The base RL algorithm for all experiments is DrQ-v2 [64]. The specific hyperparameters in Table 1.
Hyperparameters.
Observation. For most tasks, we use one 84 84 RGB image from the wrist camera as the only observation. For Tool Hang. we use a front-view camera instead since the wrist-view is heavily occluded. For Tool Hang Broad and Coffee Preparation, we use both camera views plus proprio- ception state (end-effector pose and gripper finger width). We use the default CNN structure from DrQ-v2 encode the image observations. For tasks with multiple observations, the image observations each with independent CNN network, then concatenate the CNN outputs alongside the low-dimensional observations such proprioception states to form the feature vector.
first encode
Action. All of our tasks share 7-dimensional continuous action space. It is models 6-DOF delta movement of the end-effector along with dimension for finger control. The action is modeled as normal distribution with scheduled standard deviation.
14

---

C Ablation: SPIRE without TAMP
We provide an additional ablation study on the high-level planner, TAMP. To do so, we treat the whole task as one handoff section. The agent only receives a reward one if completes the whole task. We collect 200 full demonstrations in Square, train BC policy, and apply SPIRE to fine- tune the BC policy. Since the trajectory becomes longer and the robot needs to handle object transportation, single local wrist-view becomes insufficient. We thus include both the wrist view and the global front view, as well as the robot proprioception states in the observation for the w/o TAMP variant. The result is shown Table 2.
Table 2: Comparing the success rates of Square and Square Broad with and without TAMP.
Task
BC
RL
Ours
Square w/ TAMP
98%
100%
100%
Square w/o TAMP
2%
0%
94%
Square Broad w/ TAMP
100%
100%
100%
Square Broad w/o TAMP
0%
0%
0%
Even though the w/o TAMP variant has more information from observations, the BC and RL policies are significantly worse than the w/ TAMP counterpart. The increased horizon makes the BC policy easier to drift away to regions less frequently visited in demonstrations and makes RL exploration much harder. In Square, despite the low starting quality, SPIRE still fine-tunes BC to reach a 94% success rate, demonstrating the effectiveness RL fine-tuning. However, when the initialization range increases in Square Broad, even SPIRE fails find acceptable policy.
In conclusion, TAMP (1) confines the agent-controlled section to small local area, reducing the need for global information, and (2) decreases the horizon (11.6 w/ TAMP, 101.7 w/o TAMP in Square) for the learned agent, reducing compounding errors and exploration difficulty.
15

---

D Comparison to Additional Methods
In each handoff section from TAMP, SPIRE utilizes demonstrations by training a behavior cloning agent and using RL to fine-tune There are alternative methods to combine expert demon- strations and RL, which can be readily plugged in replacements SPIRE. In this section, make connections from method GAIL [46]. The discriminator- based IRL reward in GAIL serves the same purpose our KI. penalty term preventing the current policy from deviating from the expert policy. We draw further connection by showing that our KL penalty is the same as the IRL reward function in GAIL with an alternative discriminator objective and a different reward form.
Let Te be the expert policy. The IRL reward function GAIL log (ss.a)) where D S A [0, 1] is the discriminator that maximizes
J(D) == E. log D a))l E (log(
we use an alternative objective:
J(D) Eosd (ss a)] Er (oog(Dss a))]
The alternative objective discriminates Tæ from fixed policy rather than the current learned policy т. Assume TH has full support, then maximizing (D) is equivalent to maximize for every E S:
(D) Enrn- a)] + (lyneet))loo(((( a))]
(1)
( (((),ada)) (/ TE(a log((() a)) da
(2)
( D(s,a)da) + (STelee log 8) s)
+ (STA(C s)
D(s, a)
da
a (a
(3)
( D(s,a)
(TT(( |s)) ) + ( TT(a s)log
D(s, da
(4)
Tg(a
+ H(TE(: ((ssss)))))))))) + ( TE(a s)
D(s.a)
s a(s da
TE(a 1) -1) da
(5)
( a(s,a) da
(
- TE(a sa
(.T(( (s))
D(s, ((s,a) a) da
(6)
(- 8)) 1.
where is the entropy. (5) holds since log S x for all x > and only equates when 1, i.., (s..a) < s). Since (7) is constant, the maximum of (D) be taken when (5) equates, which means the optimal solution of ((D) is D(s, a) ( s). Our KL penalty then is equivalent to using an IRL reward of log(D(s a)) = log gg (a) s).
16

---

Sec 1: Insert Nut
Sec 1:
Sec 2:
Insert Piece 1 Insert Piece 2
Sec 1: Insert Frame
Sec 2: Hang Tool
Sec 1: Insert Pod
2: Close Lid
Sec 1: Put Mug
Sec 2: Open Lid
Sec Open Drawer
4:
Sec 5:
Insert
Close Lid
Figure 7: Handoffs per task, from top-to-bottom: Square, Three Piece, Tool Hang, Coffee, Coffee Preparation. In practice, we merge section & 2, section 4 & 5 in Coffee and Coffee Preparation, there are no TAMP actions between.
E Tasks
Square and Square Broad. The robot must pick up nut and place it onto peg. This task has handoff section, where the learned agent places the nut. The Broad version increases the initialization range of both the nut and the peg.
Three Piece and Three Piece Broad. The robot must insert one piece into a base and place another piece on top of the first. This task has handoff sections, where the learned agent places the two pieces. The Broad version increases the initialization range of all three pieces including the base.
Tool Hang and Tool Hang Broad. The robot must first insert a L-shaped piece into a base to assemble a frame, then hang wrench off of the frame. This task has 2 handoff sections, where the learned agent inserts the L-shaped piece and hangs the wrench. The Broad version increases the initialization range of three pieces (base, L-shaped hook, and wrench).
17

---

Coffee and Coffee Broad. The robot must pick up coffee pod, insert it into a coffee machine, and close the lid. This task has handoff section where the learned agent inserts the pod and closes the lid. The Broad version increases the initialization range of the pod and the coffee machine.
Coffee Preparation. This is extended version of Coffee. The robot must place mug onto the coffee machine, open the lid, open the drawer with the coffee pod, pick up the pod, insert the pod into the coffee machine, and close the lid. This task has 3 handoff sections where the learned agent (1) places the mug and opens the lid, (2) opens the drawer, and (3) inserts the pod and closes the lid.
See Figure 7 for illustration of all the handoff sections.
18

---

Variance Across Seeds
In Figure 3, we show the best run out of 5 seeds. Here we provide the mean and standard deviation of the success rates in Table 3. We observe that although SPIRE still outperforms BC in terms of mean success rate in most of the tasks, our method exhibits unusually high variances in some of the tasks, for example, Coffee. Three Piece, and Tool Hang. In those tasks, one more runs result in performance significantly lower than the rest. Specifically,
In Coffee, one run has 40% success rate, while the rest are all 100%;
In Three Piece, one run has 22% success rate, while the rest are at least 98%;
In Tool Hang, one run has 0% success rate and one has 6%, while the rest are at least 82%.
Table Mean and standard deviation parenthesis) of success rales out of seeds.
Task
BC
RL [15]
Ours
Square
92.4 (5.5)
83.6 (36.7)
99.2 (1.8)
Square Broad
96.4 (4.1)
100.0 (0.0)
96.4 (5.4)
Coffee
96.8 (4.1)
40.0 (52.1)
88.0 (26.8)
Coffee Broad
41.6 (6.7)
23.2 (12.1)
84.4 (8.3)
Three Piece
63.6 (6.7)
0.0 (0.0)
84.0 (34.7)
Three Piece Broad
25.2 (7.7)
(0.0)
78.4 (5.0)
Tool Hang
9.2 (4.6)
0.0 (0.0)
54.0 (46.8)
Reinforcement learning methods are known have high variances, especially in sparse reward settings. SPIRE partially alleviates this problem by enforcing the KL penalty for deviating from an anchor policy. However, in practice, such deviation can still happen.
Figure compares the training curve of a successful run (with 88% final success rate) and failed run (with 0% final success rate). The policy in the failed run drastically deviated from the BC policy early on in the training. This is likely related to the unusually large policy gradient loss, which the KL penalty term was unable to match and failed constrain the policy.
0.03
005
-00
500
23002
41822
6206
0008
3M
2008
400%
0006
5008
3M
0
200k
-020K
000R
200k
1M
(a) Deviation from BC
(b) Policy gradient loss
(c) Reward
Figure 8: Comparing the (a) Deviation from BC, (b) policy gradient loss, and (c) reward training curves of a successful run (marked as grey) and failed run (marked as red) in Tool Hang.
In our experiments, such an abrupt decrease in policy gradient loss happens frequently, with varying scales and timing, causing the training results to have high variance. Using an adaptive weight of the KI. penalty might be potential solution, which we wish to investigate in future work.
We do not believe seeds are enough quantitatively reflect the chance of such sudden deviation happening. An alternative solution would be to compare only the results where such deviation did not happen, which is why we chose to report the top-1 performing seed in main paper.
19

---

G TAMP Formulation
We specify TAMP formulations manner similar to that Mandlekar et al. [14]. We'll use "Tool Hang" task in Figure as an example of integrating traditional and learned actions within TAMP, The goal of this task is pick and insert the frame into the stand and then pick and hang the wrench the frame. Accordingly, sequence of discrete action types that accomplishes this goal is:
a= in.ve pick move insert(...). move(..). move hang(..))]
We wish learn the most contact-rich actions, namely insert and hang. For brevity, we'll describe just the insert action since the hang action has similar description.
Like in [58], we represent planning state variables using predicates. Planning actions can be applied to planning states if they satisfy an action's of predicate preconditions (pre:). Upon application, an action modifies the truth value of the state variables through its predicate effects (efft).. Predicates and actions are parameterized by list of typed arguments. Our TAMP domain involves the following types:
conf robot configuration,
traj a robot trajectory comprised of a sequence configurations,
obj manipulable object,
grasp object grasp pose,
pose object placement pose,
policy learned closed-loop robot policy,
and the following predicates:
AtConf(g: conf) the robot currently at configuration q.
HandEmpty() the robot's hand is currently empty,
AtPose(o: obj. pose) object o is currently at placement pose p.
AtGrasp(o: obj. gi grasp) object is currently grasped with grasp pose g.
Inserted(o: obj; policy) object is inserted into the stand as result of executing policy n,
Motion(q1: conf, T: traj, 92 conf) T is a trajectory that connects configurations Q and d2-
Kin(y: conf, o: obj, y grasp, pt pose) configuration q satisfies kinematics constraint with placement pose p when object o grasped with grasp pose g.
PreInsert(o: obj, 9: grasp, q: conf, n: policy) object o grasped with grasp pose and the robot at configuration are initiation states for policy म.
The move action is deployed using a joint-trajectory controller. parameters are the current robot configuration qı, new configuration 42, and trajectory T.
move(q conf, Q2 conf, T traj)
pre: (Motion(q,7..9)) AtConf (g1)}
eff: (AtConf(q)), -AtConf
The pick action instantancously grasps object o. Its parameters are object o, future grasp pose o, current placement pose p. and current robot configuration q.
pick(o: obj, grasp, p: pose, !: conf)
pre: (Kin(q.o.g.p). AtPose(o,p). HandEmpty(), AtConf(g) eff: (AtGrasplo.9), -AtPose(o.p), -HandEmpty()}
20

---

The insert action is deployed using learned policy т. Its parameters are object current ob- ject grasp current rohot configuration (1, future configuration 12, and learned policy и. The PreInsert(o, 9, Q1. x) precondition models the set of combined object and robot states that initiate action, thus defining the initial state distribution of MDP for policy и. The Inserted(o, T) effect models the set of terminal states of the MDP for policy и, namely where the sparse reward rss) 1. Unlike move and pick, insert stochastic action because end configuration 92 de- pends on the execution of policy Because of this, we replan after executing each learned action.
insert(o: obj, g grasp, q1 conf, l2 conf, т policy) pre: (PreInsert(o, qI,T), AtGrasp(o.g). AtConf (g1)3
eff: (Inserted(o, R)), HandEmpty(), AtConf(@2) -AtGrasp(o.g), -AtConf (q))
How do we select the handoff actions?
SPIRE offers flexibility to human modelers for deciding which skills should learned, depending on the precision requirements of the interaction. In our experiments, we used our understanding of the limits of the TAMP system determine which skills should be learned. The hand-crafted skills, or the classical skills in our work include moving without colliding (transit and transfer motion) and grasping, which are fairly easy to automate. The rest of the skills are learned.
21

---

H Bridging TAMP Planner and RL
In this section, we describe how we bridge the gap between TAMP planner and RL in full detail. We follow our main text and use S and A to denote the state and action space respectively, with T: S A + S being the state transition function and S. C S being the set of goal states.
H.1 Planner Formulation
We use TAMP as our default planner in our main text. In this section, we generalize our method to any planner with the following assumptions.
A SPIRE-compatible planner P. conditioned a generated high-level plan, can be defined tuple Sia, (Spre): ( where N is the number of handoff sections; Sant S S is set of the admissible initial states; for handoff section i, Spre S the set of states where the preconditions for this section are satisfied, while Sefs S is the set where the desired effects are realized. The planner works by iterating through the sections from N. The planner should satisfy the following properties.
Sequence validity. For each section i, if the desired effects of the previous section are satisfied, i.e., Si-1 the current environment state E Seff (let S = Smit for notational convenience), the planner then plans a sequence of actions that always terminates and leads to a new state that satisfies the preconditions of the current section, i.e., E S
Section validity. For each section i, starting from any state E Spree a state s' E Skys is reachable within a finite number of steps.
Goal validity. Any state that satisfies the effects of the final section should be goal state, i.e., SN S S..
H.2
Connecting the Planner with RL
SPIRE runs by interleavingly executing the planner P and the agent iterating through the N handoff sections. In the i-th section, the current state & Styj then we terminate the process and report failure; otherwise, we execute the planner until the preconditions of this section are met. The planner then hands it off to the agent until the desired effects are met or a specified step limit has been reached.
Theorem 1. Ler P be a SPIRE-compatible planner, i.e., P satisfies sequence validity, section valid- ity, and goal validity. There exists an agent n, with which SPIRE reaches the goal state determinis- tically within a finite number of steps if the initial state is admissible by P, iee. 80 E Sinit and with large enough step limit.
Proof. With an induction on the handoff sections, we can conclude that given sequence validity and section validity there exists an agent т. with which SPIRE always reaches state S.N. within finite number of steps in each section. Given goal validity, the final state is always goal state. O
Planner-induced MDPs. In Sec 3.1, we formulate the planner-induced MDPs as a series of N MDPs with shared dynamics and different reward functions and initial state distributions, each cor- responding to planner handoff section. We can now formally define each MDP's reward function: For the i-th MDP, the reward function rs)) == 1 (eest The support of the initial state distri-
bution satisfies supp(pt) C Spre.
Sample collection rate issue. One critical issue we encountered when integrating a TAMP planner into RL training was the dropped rate sample collection. To give more context, suppose it takes a fixed time of & to sample one frame of environment interaction with an RI. agent. Under this as- sumption, the natural sampling speed with pure RL 1/t frames per second (FPS). When combined with TAMP planner, let the planning time before each handoff section be T, and let the RL agent run for at least H steps in each section, the resulting sampling speed becomes at most H/(T + tH)
22

---

t"H
Planning
Agent
(a) Single TAMP-RL. worker
(b) Parallelized TAMP-RL workers
Figure 9: Demonstrations of single-process TAMP-RL pipeline and pulti-process TAMP-RL pipeline.
FPS (see Figure 9a), relative slowdown of H1/T. In practice, the TAMP planning time T can be much longer than tHH making this slowdown significant.
Initial state distribution issue. We know that the support of the initial state distribution in the i-th section within S² However, even if we assume the planner is fixed, the distribution can still change based the agent behaviors in the previous sections. In the case of training single agent for all sections, the distribution is even more prone to policy changes the completion rates of previous sections can affect the appearance rates of later sections.
H.3 Multi-Worker Scheduling Framework
To resolve the two aforementioned issues, we designed the multi-worker scheduling framework that utilizes parallelization, as introduced in Sec 3.5.
Solving the sample collection rate issue. See Figure 9b. The multi-worker system runs TAMP planners in parallel. When a worker finishes planning and reaches the next handoff section, it will notify the scheduler to be available and enter hiatus, until selected by the scheduler for RL interac- tions. If the system has enough workers, specifically when n Z EH there always will be a TAMP worker available for RL and reach the optimal sample collection rate of 1/t FPS.
Solving the initial state distribution issue. We attempt to alleviate the changing initial state distri- bution issue by applying selection strategy to the scheduler. Specifically, the sequential strategy only accepts section when the success rate of passing all the previous sections reaches a thresh- old, which indicates that the agent's behavior in previous sections becomes stable. The framework allows for other potentially more sophisticated selection strategies, even though we not find necessary in the scope of work.
23

---

I Ablation: SPIRE without Multi-Worker
We provide an additional ablation study on how the sample collection rate boost from the multi- worker system benefits the overall performance. We run SPIRE with 16 workers and worker respectively for the same amount of 6-hour wall-clock time, for the first insertion of the Tool Hang task. The result is shown in Table 4.
Table 4: Comparing suceess rates of the first insertion in Tool Hang with and without multiple workers.
# Workers
Success rate
Sampling FPS
16
98%
55.4
1
42%
7.7
24

---

J Additional Experiment Results
We show the RL learning curves Figure 10 comparing SPIRE and naive RL. The x-axis represents the environment steps and the y-axis represents the success rate.
0.8
0.8
0.6
0.6
0.2
0.2
200k
400k
600k
800k
1M
200k
600k
800k
4
(a) Square
(b) Square Broad
0.8
0.8
0.6
0.6
0.2
0.2
200k
400k
600k
800K
1M
200k
400K
600k
800k
1M
Coffee
(d) Coffee Broad
0.8
0.8
MW
0.6
0.4
0.4
0.2
0.2
200k
400k
600k
800k
1M
0
200k
400k
600k
800k
1M
Three Piece
(l) Three Piece Broad
Figure 10: Comparing the RL learning curves of SPIRE and naive RL.
25


Reviews:
[{"review_id": "H6sv1TJBEZ", "reviewer": "Reviewer_bC4r", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "A framework based on TAMP utilizing BC and RL.", "summary_of_recommendation": "Good paper, but nothing really outstanding to be accepted. Seems very incremental with many assumtions that need to be discussed.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "Questions:\n\n- Which skills are learned and which are hand-crafted? What are \"classical skills\"?\n- Where does the TAMP model come from?\n- How is it decided which tasks are handed over?\n- How are symbolic predicates obtained?\n- Which planner is used?\n- In the pipeline for handing over, is the complete plan from TAMP available?\n- TAMP planner seems to use cover part but it is not explained.\n- Why is RL failing on those domains?\n- How does your method generalize to novel domains?\n- Learning curve for RL and SPIRE would be very informative.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VjyZ7G4bhV", "reviewer": "Reviewer_TuDy", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a hybrid learning method aimed at long-horizon manipulation tasks. Within the framework, the TAMP layer executes manipulation for plannable actions. For non-plannable actions, it enters handoff sections. For training, human demonstration data is collected in a human-in-the-loop manner in the handoff sections. Then, policies are initially trained on the demonstration data through behavior cloning and later fine-tuned by RL. During testing, the TAMP planner and these handoff policies are used for manipulation tasks.", "summary_of_recommendation": "I think the idea of integrating TAMP, IL, and RL in this paper is interesting. Additionally, the proposed framework is sound, and the results support the strength of the method. However, there are some unclear parts regarding SPIRE’s TAMP layer, as mentioned in my review.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "- The proposed method uses TAMP. Therefore, it would be worthwhile to add a vanilla TAMP baseline that does not use the gated RL/BC policy. This would highlight the strength of SPIRE over the TAMP planner alone. If this is not available, is there any way to adapt it for a baseline?\n- Is the TAMP used in SPIRE model-based? If so, it might require full geometric information of the environment. However, it seems that the observation space is provided as visual observation from multiple viewpoints. Doesn't this cause partial observability for TAMP?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZwCRLRRbne", "reviewer": "Reviewer_doj6", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The key idea of this paper is that combining behavior cloning (BC) and RL fine-tuning can lead to completing tasks more efficiently and reliably within a hybrid TAMP system that uses a combination of learned and non-learned skills. Authors state a secondary contribution of suggested techniques for efficient RL finetuning from a BC prior. Experiments show large improvements in success rates (reliability), episode length (efficiency) and demo data-efficiency.", "summary_of_recommendation": "Currently, the (main) paper does not clearly communicate the authors ideas about integrating BC-initialized RL policies into their TAMP system. After reading author responses, I can see why the BC/IL addition is a nontrivial and helpful addition to HITL-TAMP. My scores are updated to reflect a better understanding of the technical contribution. I am leaving my recommendation as \"Weak Reject\" since I think the paper needs major revision beyond the revised manuscript.  I recommend to the authors that they re-organize the paper to highlight the most significant ideas in the main paper.", "technical_quality": 4, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Please clarify the significance of adding IL initialized / RL fine-tuned policies that is not shown by [14] or [15] What are the challenges and added complexities of finetuning IL policies to be used in a TAMP system that do not apply to BC policies or RL policies in general?\n  \n\n2. What TAMP method do you use? At the very least, what is needed from a TAMP system to be compatible with this method? It seems to me implicitly that the TAMP method needs to be [14]. Ex) how are goals specified, how is state information used. A PDDL file in the supplementary would help.\n\n3. Related to above, where do the postconditions come from and how are they used during both IL and RL? Does a demonstration need to satisfy the postcondition? How does the demonstrator know what to do? \n\n4. If you have the preconditions and postconditions and a simulated environment, why do you need to run the TAMP planner to gather the dataset for learned skills?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper combines TAMP, behavior cloning, and RL, to built a system that can solve complex long-horizon manipulation problems.

Although on the first sight this work can be seen as a mere combination of TAMP, BC, and RL, this paper is relevant for the community because it bridges the gap between TAMP that often cannot deal with manipulation tasks that require high precision, and RL which struggles with long horizon tasks. This combination and the insights are novel, and hence of value for the community.

The authors provided a strong rebuttal addressing most of the concerns of the reviewers.

Please note that according to one reviewer, the paper still need further major revision for clarity and presentation.
Decision: Accept