Jacta: A Versatile Planner for Learning Dexterous and Whole-body Manipulation
Authors: Jan Bruedigam
Ali Adeeb Abbas
Maks Sorokin
Kuan Fang
Brandon Hung
Maya Guru
Stefan Georg Sosnowski
Jiuguang Wang
Sandra Hirche
Simon Le Cleac'h
Venue: CoRL 2024 (2024)
Keywords: Dexterous Manipulation Planning
Learning with Demonstrations
TL;DR: We introduce a manipulation planner that enables bootstrapped reinforcement learning of dexterous and whole-body manipulation tasks.

Abstract: Robotic manipulation is challenging due to discontinuous dynamics, as well as high-dimensional state and action spaces. Data-driven approaches that succeed in manipulation tasks require large amounts of data and expert demonstrations, typically from humans. Existing planners are restricted to specific systems and often depend on specialized algorithms for using demonstrations. Therefore, we introduce a flexible motion planner tailored to dexterous and whole-body manipulation tasks. Our planner creates readily usable demonstrations for reinforcement learning algorithms, eliminating the need for additional training pipeline complexities. With this approach, we can efficiently learn policies for complex manipulation tasks, where traditional reinforcement learning alone only makes little progress. Furthermore, we demonstrate that learned policies are transferable to real robotic systems for solving complex dexterous manipulation tasks.

Project website: https://jacta-manipulation.github.io/

Jacta: A Versatile Planner for Learning Dexterous and Whole-body Manipulation
Jan Brüdigam', Ali-Adeeb Abbas?, Maks Sorokin', Kuan Fang', Brandon Hung", Maya Guru², Stefan Sosnowski', Jiuguang Wang', Sandra Hirche', Simon Le Cleac'h
TU Munich', Boston Dynamics Al Institute?
Abstract: Robotic manipulation is challenging due discontinuous dynamics, well high-dimensional state and action spaces. Data-driven approaches that succeed in manipulation tasks require large amounts of data and expert demonstra- tions, typically from humans. Existing planners are restricted to specific systems and often depend on specialized algorithms for using demonstrations. Therefore, we introduce a flexible motion planner tailored to dexterous and whole-body ma- nipulation tasks. Our planner creates readily usable demonstrations for reinforce- ment learning algorithms, eliminating the need for additional training pipeline complexities. With this approach, we can efficiently learn policies for complex manipulation tasks, where traditional reinforcement learning alone only makes little progress. Furthermore, we demonstrate that learned policies are transferable to real robotic systems for solving complex dexterous manipulation tasks. wew.ggittu...... io/
Project website: https://jacct--antt
Keywords: Dexterous Manipulation Planning, Learning with Demonstrations
Introduction
Dexterous and whole-body robotic manipulation in high-dimensional state-action space with non- smooth dynamics is challenging, but learning-based approaches have shown encouraging results for such tasks. However, they require loads of high-quality data [1, 2] extensive exploration to find solutions. Yet, robot data is scarce and expensive; human demonstrations are costly and confined to human capabilities. Moreover, using dense rewards or curriculum learning to guide the policy search potentially imposes a sub-optimal solution structure and necessitates lengthy reward shaping.
promising solution these issues is to leverage motion planners to automatically generate demon- strations for learning instead of relying humans [3, 4, 5). This approach addresses the challenge of scaling data collection. However, existing manipulation planners are often designed for specific end effectors and single tasks such pushing [6, 7]. pick-and-place, [8, 9, 10] or in-hand manipu- lation [11, 12]. Further, using the demonstrations generated by these motion planners is non-trivial because they assume quasi-static dynamics [13, 14, 15] or only explicitly provide state trajectories [12]. This paper shows that robotic manipulation must not be limited in this manner. planner can combine highly dexterous capabilities precisely manipulate small items and whole-body interac- tions to maneuver large objects, with contact occurring not just the end effector.
To address these issues, we introduce novel planner and learning pipeline for dexterous and whole- body robotic manipulation, shown in Fig. 1. The planner finds solutions to manipulation tasks through sampling-base based and gradient- not limited to quasi-static systems, nor does need to explicitly reason about the contact modes. The aim is not use the planner directly in receding horizon fashion akin model predictive control but to leverage daanerr-gnerran demonstrations bootstrap policy learning. Thus, when using the planner, one is not bound to human demonstrations and human-like robotic systems. Instead, one
actions for global and goal-directed exploration. It is
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Box graep
start:
80
Manipulation Planner
PILASE
dense
2.2
81
actions
rewards
Environment
PILASE
85
sparse
rewards
Reinforcement
Learning
86
goal:
N
halanced
Roplay
Stool
Ivers
components
sompling
random
peaxtmity
proximity
continuztion
B.5
reachability
goal-direction
Our manipulation planner generates a search tree of dynamically feasible state-action trajectories. During simulation training, planner trajectories are fed to the replay buffer of rein- forcement learning algorithm. The resulting simulation-trained policies are deployed on real robotic systems for different manipulation tasks.
Figure
can build on learning methods that are bootstrapped with the planner's solutions. Surprisingly, we find that simply adding a small number of planner demonstrations to the replay buffer enough to drastically improve learning performance and generate successful policies for complex tasks.
In summary, our contributions are the following:
Design of novel motion planner generating dynamically feasible state-action trajectories for dexterous and whole-body manipulation tasks.
Identification simple yet efficient learning pipeline for using the planner's demonstra- tions in reinforcement learning.
Transfer of trained policies to real systems to demonstrate the feasibility of complex tasks covering both dexterous and whole-body manipulation.
2
Related work
Guiding Manipulation Learning with Demonstrations Data-driven approaches are state-of-the- art for solving complex manipulation tasks. Imitation learning (II.) leverages expert demonstrations of such tasks train policies for robots. Recent results show the great potential of IL in advanced tasks, such as unseen object manipulation [16. 17], door opening [18], manipulation of liquids and fabrics [19]. Yet, IL requires large amounts of expert demonstrations to deliver working policies, which bottleneck for quickly learning new skills, especially since human demonstrations are inherently limited to human capabilities and costly to collect. Reinforcement learning (RL) does not necessarily require demonstrations due to its noisy exploration. However, the initially random exploration in large state-action spaces has a low chance of success. Accordingly, demonstrations have also been used to guide RL with additional offline data. Two different ways of incorporating demonstrations into RL have found popularity: directly adding the demonstrations to the replay buffer with minor modifications to the underlying learning algorithm [20, 21, 22] or pre-training an imitation policy from the demonstrations [23, 24, 25J. Still, the advanced manipulation tasks investigated in [23, 24] only cover anthropomorphic systems and require human demonstrations. In contrast, we propose a pipeline with planner that is not limited to anthropomorphic systems and eliminates the need for costly human data collection, thus offering greater scalability.
Planning for Robotic Manipulation Learning While robotic motion planning has seen great success, especially with sampling-based methods [26. 27], manipulation planning remains partic- ularly difficult due to the multitude of contacts in general settings. Thus, most methods focus motion planning for low-dextyyyty grippers and specialized tasks such as pushing fixed grasping

---

[6, 7, 8, 28, 29]. More advanced techniques plan dexterous manipulation assuming quasi-static con- tact dynamics [15, 14] and myopic sampling-based planning offers compelling results but only on short-horizon tasks [30]. Manipulation planning based on optimal control often requires smoothed contact dynamics [31] or scheduled contact sequences [32, 33] since contact-implicit schemes are limited simple tasks in practice [34, 35, 36]. In contrast, our proposed planner designed solve dexterous and whole-body robotic manipulation tasks without restrictions specific end effectors or contact locations. We allow contact on all parts of robot and plan with full contact dynamics to generate readily usable demonstrations for manipulation learning.
Since manipulation planners are not fast enough to be directly deployed on the robot systems, they have been combined with learning methods. Collision-free motion planners have been used as an alternative to RL's random exploration, but the applications have been limited to simple manipula- tion tasks [4, 3]. Moreover, motion-planner actions do not overcome the lack of reward signals in sparse reward settings [37, 38). Demonstrations from motion planners have served as expert data for IL with pick- and--eee tasks [5] and for RL with basic reaching tasks rather than full-scale manipu- lation [39, 40]. Planner-based demonstrations for manipulation have been used to find an informed start-state distribution for the RL exploration for simple object rearrangement tasks [13], and a sim- ilar approach has been developed for dexterous in-hand manipulation [12]. However, these methods implement task-specialized planners and are not directly portable generic manipulation settings. Additionally, they only provide state trajectories without associated actions, making it difficult to use the demonstrations directly for RL. Both of these issues are addressed with our planner, which produces dynamically feasible state-action demonstrations for general robotic manipulation tasks.
3 Background
This section describes the robotic manipulation scenario solved by our motion planner and the rein- forcement learning algorithm used to optimize policies for the manipulation tasks.
Robotic manipulation scenario We consider one or multiple robots with joint configuration q, E Rd and joint velocity! 9, E IR that interact with one or multiple objects with configuration q. E g and velocity d. E R"",, These quantities form the state s = 1q. q. q. a..1" E I The actions a E IR= are position commands for the robots' actuated joints. The position commands are tracked by low-level controller (cf. Appendix A). The potentially non non-smooth dynamics for this system states. The planner's task is to find a state-action trajectory that minimizes the weighted distance d(s, $g) = ||s -glle,, to desired goal state 8g. The weight matrix Qa E R", Xm,, is used to prioritize certain dimensions of the state, e.g., of the objects, and ignore others, e.g., of the robots.
81+1 f(s,, a.) time and contain bounds the robots' and objects'
Manipulation policies are found with value-based reinforce- ment learning [41, 42]. A goal-oriented value function Q(s, a, 8g) E IR for the system's state- action-goal space is learned from rewards r. The sparse reward function r(-) E (-1, 0} is 0 if the relative distance d of state 8, to the start state 8o is below a threshold e, and otherwise:
Reinforcement learning algorithm
if d(se.se) 2(s0.=g) Se otherwise
r(s 8g)
(1)
The optimal value function given this reward function is
Q" (s,,, az 8g) = E
r(st. sg) + +
maxx 0.124,1,,411. 8g)
(2)
s.11
where y E [0, 1) is a discount factor. Approximating the value function with a network Qo enables learning the network's parameters & from tuples (s,, ae 8gz Tt, 8441) in continuous spaces. The optimal policy a T²* (s sz) maximizes the value function:
n (s, 8g) = arg max Q* a, 8g),
(3)
If' the robot state contains quaternion, the joint velocity is lower dimensional than the joint contiguration.
3

---

and is also approximated by a network To with parameters 6, which provides continuous policy representation. Since we want to use demonstrations from our motion planner for learning, we use the off-policy learning algorithm deep deterministic policy gradients (DDPG) [41].
Dexterous and whole-body manipulation planner
We design motion planner generate dynamically feasible state-action trajecto- ries for dexterous and whole-body manip- ulation. These solutions serve as demon- strations to bootstrap policy learning. The planner builds a tree starting from a sin- gle start state by taking actions towards the goal. For each action taken, a new node is added to the tree, as shown in Fig. 1. In this way, the search space is explored until a viable trajectory to the goal is found.
Algorithm 1 Manipulation Planner
1: Choose Tlg. 74j 3, fee Pa. At, 81; Set An
2: for ig in 1 Rig do
3:
gow....saaaal..... sampl ing(hg Smin, Smax) ) Sec. 4.1 4: reward.updatef
b Sec. 4.2
5: for i; in m do
Re node www.eteeeeeenne...e no)
Sec. 4.3
7:
for in in : The do
8:
an action.sampling(//
Sec. 4.4 >> Sec. 4.5
9:
8n,+1 extension n,,, an.+1)
10:
nn nh +
11:
in Pn
Robotic manipulation has discontinuous contact dynamics and rugged optimization landscapes [43]. Thus. the planner involves several com- ponents. Gradient-bart based actions for locally optimal search combined with random actions for global exploration and avoiding local minima. Reward heuristics are used to make progress in high- dimensional spaces. The resulting planner is applicable to general robots and manipulation tasks. We only assume the availability of a simulator for dynamics and gradients, e.g., Mujoco [44]. The search algorithm is stated in Alg. 1, and the components the planner are introduced in the subsec- tions below. Ablations for the planner's parameters are given in Appendix C.
4.1 Goal sampling
Manipulation settings have highly constrained non-convex search spaces with many local minima. Inspired by rapidly exploring random trees [26]. ng different goal states are sampled during the search to escape local minima and enable global exploration. bias bg E [0, 1] is set to sample with a certain probability the actual goal for a given task ttakk Otherwise, a random state sampled
uniformly a box 8random
E [8min; 8 with upper and lower bounds 8min and 8max!
&task
if Pg S bg Srandom otherwise
8g
with Ps ~ U(0, 1), 8random ui (min: 9max).
(4)
Sampled non- unit-length orientation representations. i.e., quaternions, are normalized. Infeasible states, i.e., states with contact penetration, are discarded and resampled.
4.2 Reward metric
Exploration in high-dimensional spaces challenging due the combinatorial explosion. Thus, each node in the tree is assigned value to permit heuristic selection of promising nodes during the search, similar to Monte Carlo tree search [45]. Instead of a discounted value, directly using node's ' reward yields satisfactory results for us and removes the need for value back prokppropagation up the tree. When the goal state 8g is changed, the total reward (value) of a node with state 8,
r(s, 8g) = ra (8,8 82) + (s) + (8, is computed from the three following reward components.
(5)
Distance The actual closeness to the desired manipulation goal is formulated as a distance reward,
(s -||s ass-slllg.
with weight matrix Qa E Xn, It encourages small-scaled distance between node's state and the goal. Note that for quaternions, the three-dimensional orientation difference is used [46].

---

Proximity For manipulation, robot-object interactions must occur. Such interactions are more likely if the robot is close to the object and a reward is given for such states. The distance between robots and objects is measured by "p virtual proximity sensors. Sensor i E {o...., np)) consists of a pair of points with Pr, attached to the robot hardware and Po.: attached to the object. returns a distance d, lip.. Po.ill- The proximity reward for a node
(d(d) - -lddll,,
where Qp E P a weight matrix and d = [dl,.... dnp.]' E IR consists of all stacked sensor values. Close proximity between the robot and the object ensures frequent whole-body in- teractions but may prevent exploration because moving away from the object change the robot's configuration is penalized. The weight matrix Qp is responsible for achieving good trade-off.
Reachability When being close to an object, a robot needs to be in configuration where moving the object towards the goal is casily possible. Given an object's state difference from the goal As. = 8. - 8g.o- a measure m for the difficulty of reaching the goal can be calculated with the local Mahalanobis metric defined [14] as
m M.. with M (B.B.. + "o B. oS. E
da
Here, B. is the control Jacobian of the object dynamics f. with respect to the action P E R is regularization parameter to ensure the invertibility of M, and I E R the identity matrix. A small m indicates that moving the object towards the goal easy in the current robot configuration. The reachability reward based on m is defined as
I'm (m) Im log(m),
(9)
with weight gm. In practice, m can be clipped to avoid overly large rewards (cf. Appendix A).
4.3
Node selection and extension horizon
At the beginning of each of the n; iterations, node with index la N4 must be selected. A trade- off is required between selecting the currently best node for greedy exploitation and selecting worse nodes for exploration. To this end, the tree's n, nodes are ranked from highest to lowest reward, and the index in = [x ~ P( B)] is the rounded sample from a truncated Pareto distribution [47] with bounds [1, n., and exponent B E R4. Larger values for the parameter 3 lead to a more greedy node selection. After the first node is selected, given the extension horizon ne, the remaining n. node indices for the current iteration are the last extended nodes' indices. In practice, and ne can be varied when the search is progressing well or poorly (cf. Appendix A).
4.4 Actions
Exploration is based on mix of globally exploring random actions to avoid local minima and locally optimal actions for precise fine-grained manipulation. Once a node is selected, step size Al, and an action a âoœ is generated with element- wise multiplication of the direction and magnitude E [TR"> |
{R" || = 1} Ommax) ), where Ommax is the maximum magnitude. The action corresponds to relative position command and is calculated using one of the methods listed below. The method used is sampled according to user-specified discrete probability distribution p.. In practice, the step size Ats for each action can vary (cf. Appendix A).
Random To ensure global exploration, random action can be taken. This action samples uniformly distributed direction and the magnitude is sampled uniformly within the magnitude bounds.
Continuation Obtaining a promising search direction is challenging, so instead of discarding a successful action after one step, it can prove beneficial keep going in this direction. Accordingly, the continuation action takes the same action direction that leads the currently selected node, i.e.. à;, a where and ju are the current and parent node indices, respectively. The magnitude a is sampled uniformly within the magnitude bounds.
5

---

Proximity As explained for the proximity reward, closeness of robot and object is necessary for manipulation. Therefore, the values of the virtual proximity sensors are also used to compute the proximity action. The action direction aims to minimize d d and points in the opposite direction of the gradient g = (ōd/aa) d. i.e., à 10g//gl9 ||gll- The magnitude a is sampled uniformly within the magnitude bounds.
Goal-direction Besides random actions for global exploration, coal-directed actions are required to make meaningful progress. Locally-opiimized actions are computed by linearizing the dynamics around the reference values si = so and aj = ag - Aag. with the fixed reference state 8o:
0f
so ao) fö Bda., with fü f i ag), B
RR
(10)
da aj
The linearized dynamics are used solve the one-step optimal control problem 1 Adg 2 (81 8g) Q(1 8g) + (ap Aaa) R (aj + Aao)
min 1
(11a)
s.t. 81 = f + Bdn0,
(11b)
with closed-form solution for Dag (ef. Appendix A). The resulting action direction is à (aj + A A where aj can be chosen as, for example, the action used reach the current node. The action magnitude is set to the maximum value, i.c., Qmax.
4.5 Tree extension
Once the action computed for a selected node, a dynamics rollout is performed. The reached state appended to the search tree new node. Details on the rollout are given in Appendix A.
5
Bootstrapping reinforcement learning with demonstrations
Our planner can find solutions for challenging manipulation tasks but provides demonstrations only for a single start state. Accordingly, we do not deploy the planner directly as an online policy but instead use the demonstrations to bootstrap reinforcement Learning (RL). This approach allows us generalize beyond the planner's demonstrations and perform additional online exploration to exceed the planner's suboptimal performance stemming from partially random actions. By distilling policy, also longer require the planner during policy training or execution. For the best inclusion of demonstration, we investigate several different methods, which can be grouped into the two main strategies outlined below. Further details are provided in Appendix B. Foreshadowing the results, we find that using fixed ratio of demonstrations in the replay buffer provides the best performance of investigated methods. This result aligns with the core idea off-policy RL [48] and findings in [22]. a welcoming result due to the simplicity of implementing this strategy.
5.1 Filling the replay buffer with demonstrations
During each training epoch, Πp + rollouts, consisting of np offline planner demonstrations and nt online policy rollouts, are appended to the replay buffer. Afterward, training data is sampled randomly from the entire replay buffer. Similar to [22], we investigate three different approaches for filling the replay buffer: (1) using fixed ratio np/n,, n/,,, each epoch, (2) using decreasing ratio as the policy's 's success rate increases, and (3) adding only planner demonstrations in the first epoch and only online policy rollouts afterward.
5.2 Pre-training policies and value functions
Following the concepts outlined in [25], we investigate three different approaches for using demon- strations for pre-training: (1) pre-training an imitation policy TII from demonstrations and using as fixed policy alongside the RL-trained policy TRL, (2) pre-training मा, and initializing the value function Q from demonstrations, and (3) pre-training ail as well as initializing Q and additionally adding fixed ratio of demonstrations to the replay buffer as described above.

---

6 Experiments
Given our motion planning and learning setup, we investigate three main questions:
A) Can the planner generate solutions for dexterous and whole-body manipulation tasks?
B) Can the solution demonstrations bootstrap manipulation learning?
C) Can the learned policies be transferred to real systems?
These questions are evaluated for the six tasks displayed in Fig. and Note that additional tasks, baseline comparisons, and ablations, as well details systems, parameters, and experiment setups, are given in Appendix C. For each evaluation run, five random seeds are used.
A) Manipulation planning The planner's ability to solve the manipulation tasks is measured as the search progress = 1 -
/r) (So sg). i.e., the scaled relative closeness to the goal (cf. (6)). This progress is plotted for the six tasks in Fig. 2. Despite the varying tasks and complexity of the settings, the planner consistently finds solutions for all scenarios, demonstrating that the rewards and actions are general enough to treat high-dimensional spaces and multi-contact dynamics.
(s,, TTa
search pearch progress
1.0
Planar hand
1.0
Floating hand
1.0
Allegro hand
Planor hand
Allegre hand
0.5
0.5
0.5
0.0
0.0
5000
5000
5000
Box grasp
Ball turn
Stool lift
Floating hand
search progress
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
5000
5000
5000
simulation steps
simulation steps
simulation steps
the performance of the Manipulation Planner (Alg. 1) on six tasks. Relative close- ness to the goal for the best node in the tree so far vs. the total number of nodes (simulation steps). Right: the three additional tasks used for evaluation besides the tasks shown in Fig. 1.
Figure 2: Left:
B) Learning with Demonstrations We compare the following approaches for demonstration in- clusion in Fig. 3: pure DDPG as a baseline, using a fixed ratio of np/n,, 25% demonstrations for the replay buffer (cf. Sec. 5.1), and pre-training an imitation policy as a secondary policy (cf. Sec. 5.2). For comparability with the planner, the learning progress is plotted over the total actions taken. Using demonstrations always leads to an improvement over pure reinforcement learning, sometimes drastically, while pre-training an imitation policy performs inconsistently. Even though the demon- strations are limited to a single start state, they provide enough reward signals to learn successful policies faster and more robustly for all runs. At the same time, the demonstrations are not good enough to directly train and use an imitation policy, as adding this policy does not yield satisfactory results as it is slower and does not make any learning progress for some of the runs.
C) Transfer to real systems We transfer policies from training in simulation directly to the hardware for the three tasks shown in Fig. 1. For each setting, we pick a policy with good performance in simulation but perform no additional training in the real world. Table shows the evaluation success rates on an initial-state grid, and Fig. 4 shows the progression of the stool-lift task an example. The trained policies for the box grasp and stool-lift tasks work well, given the direct policy deploy- ment. It is worth noting that in the stool-lift task, contact occurs the robot arms, not just the
Trials
Success
Box grasp
27
93%
Ball turn
20
50%
Stool lift
25
76%
Table 1: Success rate on real systems.
7

---

Planar hand
Floating hand
Allegro hand
1.0
1.0
1.0
slicess rale
0.5
0.5
0.5
0.0
0.0
0.0
x 10
0
1 10
0
1 10
Box grasp
Ball turn
Stool lift
1.0
1.0
success rale
0.5
0.5
0.5
0.0
1 10 simulation steps
10"
2
10"
0
x
10"
10"
simulation
simulation steps
RL - RL with pre-tnaining
RL with planner demonsttations (ours)
Figure 3: The performance of reinforcement learning for manipulation on six tasks. Comparison of our approach of adding demonstrations the replay buffer (blue), RL with additional imitation policy pre-traine trainiiinng (red), and pure RL (black). Further baseline comparisons are in Appendix C.
end effectors. Transferring the ball-turn task to the real world is more challenging due to modeling approximations, the yoga ball is simulated perfectly rigid object.
initial stool position
0.05
7.05
15.05
22.05
/
x
X
Figure 4: Left: the successful and failed initial positions for the stool-lift tasks. Right: deployment of the trained policy for two Spots lifting and placing stool in the upright position.
7 Conclusions
We present versatile planner for dexterous and whole-body manipulation tasks. The planner gener- ates demonstrations that are used to bootstrap reinforcement learning, leading to significant learning improvements. It is robust against increasing task complexity, making it a valuable tool for gen- eral and complex manipulation scenarios. Similarly, aiding reinforcement learning with synthetic demonstrations from the planner is simple to implement yet gives strong results, providing a promis- ing architecture choice for general manipulation learning, also combination with other methods.
Limitations Our method has limitations. (1) Like any automated search, the planner requires hy- perparameter tuning. However, the planner provide results within seconds compared to learning algorithms that often require minutes or hours before success can be assessed. This fast process drastically reduces the iteration loop and allows manual tuning at interactive rates and quick param- eter sweeps. (2) Both the planner and policies rely on state information, which requires motion capture system the real world to acquire such state information. There are several ways to extend this work to synthesize vision-basee based policies. One could transfer planner data to convert state-action trajectories image-action trajectories. second option would be apply a teacher-student frame- work [49, 50, 51, 52, 38] by training student policy that only has access to vision sensors mimic expert state-based policy. (3) Finally,
relatively simple pre-training setup, and the reported performance of imitation pre-training could change with more sophisticated II. methods.

---

Acknowledgments
We would like to thank Emmanuel Panov, Ciarán T. O'Neill, and Stephen Proulx for their help with the hardware experiments. We would also like to thank Dmitry Yershov, Gustavo Nunes Goretkin, Farzad Niroui, Martin Schuck, and Petar Bevanda for technical discussions and implementation help. Finally, we would like to thank Tao Pang and Tong Zhao for their help and guidance in running baseline benchmark comparisons.
This work was supported by the European Union's Horizon Europe innovation action programme under grant agreement No. 101093822, "SeaClear2.0"..
References
[1] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany. C. Wang. R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning, pages 1678-1690. PMLR, 2022.
[2] S. Belkhale, Cui, and D. Sadigh. Data quality in imitation learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, 2024.
[3] T. Tosun, E. Mitchell, B. Eisner, J. Huh, B. Lee, D. Lee, V. Isler, H. S. Seung, and D. Lee. Pixels to plans: Learning non-prehensile manipulation by imitating a planner. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7431-7438. IEEE, 2019.
[4] B. Abbatematteo, E. Rosen, S. Tellex, and G. Konidaris. Bootstrapping motor skill learning with motion planning. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4926-4933 4933. IEEE, 2021.
[5] M. Dalal, A. Mandlekar, C. Garrett, A. Handa, R. Salakhutdinov, and D. Fox. Imitating task and motion planning with visuomotor transformers. arXiv preprint arXiv:2305.16309, 2023.
[6] Scholz and M. Stilman. Combining motion plamning and optimization for flexible robot manipulation. In 2010 10th IEEE-RAS International Conference Humanoid Robots, pages 80-85. IEEE, 2010.
[7] E. Arruda, M. J. Mathew, M. Kopicki, M. Mistry, M. Azad, and L.. Wyatt. Uncertainty averse pushing with model predictive path integral control. In 20/7 IEEE-RAS 17th International Conference Humanoid Robotics (Humanoids), pages 497-502. IEEE, 2017.
[8] J. Huh, B. Lee, and D. D. Lee. Constrained sampling-bart bbsed planning for grasping and manip- ulation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 223-230. IEEE, 2018.
[9] A. Simeonov, Y. Du, B. Kim, F. Hogan, J. Tenenbaum, P. Agrawal, and A. Rodriguez. A long horizon planning framework for manipulating rigid pointcloud objects. Conference on Robot Learning, pages 1582-1601. PMLR, 2021.
[10] J. Jankowski, L. Brudermüller, N. Hawes, and S. Calinon. VP-STO: Via-point-based Stochas- tic Trajectory Optimization for Reactive Robot Behavior. In 2023 IEEE International Confer- ence on Robotics and Automation (ICRA), pages 10125-10131. IEEE, 2023.
[11] B. Sundaralingam and T. Hermans. Geometric in-hand regrasp planning: Alternating opti- mization of finger gaits and in-grasp manipulation. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 231-238. IEEE, 2018.
[12] G. Khandate, T. L. Saidi, S. Shang, E. T. Chang, Y. Liu, S. Dennis, Adams, and M. Ciocarlie. RxR: Rapid exploration for Reinforcement Learning via Sampling- based Reset Distributions and Imitation Pre-training, arXiv preprint arXiv:2401.15453 2024.

---

[13] L. Pinto, A. Mandalika, B. Hou, and S. Srinivasa. Sample-efficiet learning of nonpre- hensile manipulation policies via physics-based informed state distributions. arXiv preprint 1810v 100.:/000.0004 2018.
[14] T. Pang, Suh, L. Yang, and R. Tedrake. Global planning for contact-rich manipulation via local smoothing of quasi-dynamic contact models. IEEE Transactions on Robotics, 2023.
[15] X. Cheng, E. Huang, Y. Hou, and M. T. Mason. Contact mode guided motion planning for quasidynamic dexterous manipulation in 3d. In 2022 International Conference on Robotics and Automation (ICRA), pages 2730-2736. IEEE, 2022.
[16] E. Jang, A. Irpan, M. Khansari, D. Kappler, Ebert, C. Lynch, S. Levine, and C. Finn. BC- Z: Zero-Shot Task Generalization with Robotic Imitation Learning. Conference on Robot Learning, pages 991-1002. PMLR, 2022.
[17] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. RT-2: Vision-Language-Ae Models Transfer Web Knowledge to Robotic Control. arXiv preprint arXiv:2307.15816 2023.
[18] M. Khansari, D. Ho, Y. Du, A. Fuentes, M. Bennice, N. Sievers, S. Kirmani, Bai, and E. Jang. Practical imitation learning in the real world via task consisteney loss. arXiv preprint arXiv:2202.0188 2022.
[19] C. Chi, S. Feng. Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[20] M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint 1XX7v:7707.088 2017.
[21] A. Nair, B. McGrew, M. Andrychowicz. W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference robotics and automation (ICRA), pages 6292-6299. IEEE, 2018.
[22] J. Ball, L. Smith, I. Kostrikov, and S. Levine Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pages 1577-1594. PMLR, 2023.
[23] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. arXiv preprint
10087, 2017.
[24] N. Hansen, Y. Lin, H. Su, X. Wang, V. Kumar, and A. Rajeswaran. Modem: Accelerating visual model-based reinforcement learning with demonstrations. arXiv preprint arXiv:22/2.05694 2022.
[25] H. Hu, S. Mirchandani, and D. Sadigh. Imitation bootstrapped reinforcement learning. arXiv preprint arXiv:2311.02198, 2023.
[26] S. LaValle Rapidly-exploring random trees new tool for path planning. Research Report 9811, 1998.
[27] L. E. Kavraki, Svestka, J.-C. Latombe, and M. H. Overmars. Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE transactions on Robotics and Automation, 12(4):566-58) 1996.
[28] T. Gold, A. Völz, and K. Graichen. Model predictive interaction control for robotic manipula- tion tasks. IEEE Transactions Robotics, 39(1):76-89 2022.
10

---

[29] R. Holladay, T. Lozano-Péree and A. Rodriguez. Robust planning for stati-stage forceful manipulation. The International Journal of Robotics Research, 43(3):330-353, 2024.
[30] T. Howell, N. Gileadi, S. Tunyasuvunakool, K. Zakka, T. Erez, and Y. Tassa. Predictive sam- pling: Real-time behaviour synthesis with mujoco. arXiv preprint ar.ivv2212.005..5. 2022.
[31] R. Natarajan, G. L. Johnston, N. Simaan, M. Likhachev, and H. Choset. Torque- limited manip- ulation planning through contact by interleaving graph search and trajectory optimization. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8148-8155 -154. IEEE, 2023.
[32] J.-P. Sleiman, Farshidian, M. V. Minniti, and M. Hutter. A unified MPC framework for whole-body dynamic locomotion and manipulation. IEEE Robotics and Automation Letters, ():4688-4695, 2021.
[33] X. Cheng, S. Patil, Z. Temel, O. Kroemer, and M. T. Mason. Enhancing dexterity in robotic manipulation via hierarchical contact exploration. IEEE Robotics and Automation Letters, (1):390-397, 2023.
[34] J.-P. Sleiman, J. Carius, R. Grandia, M. Wermelinger, and M. Hutter. Contact-implicit trajec- tory optimization for dynamic object manipulation. In 2019 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), pages 6814-6821, IEEE, 2019.
[35] S. Le Cleac'h, h, T. A. Howell, S. Yang, C.-Y. Lee, J. Zhang, A. Bishop, M. Schwager, and Z. Manchester. Fast contact-implicit model predictive control. IEEE Transactions on Robotics, 2024.
[36] A. Aydinoglu and M. Posa. Real-time multi-contact model predictive control via admm. In 2022 International Conference on Robotics and Automation (ICRA), pages 3414-3421, IEEE, 2022.
[37] J. Yamada, Y. Lee, G. Salhotra, K. Pertsch, M. Pflueger, G. Sukhatme, J. Lim, and P. En- glert. Motion planner augmented reinforcement learning for robot manipulation in obstructed environments. In Conference on Robot Learning, pages 589-603. PMLR, 2021.
[38] I.C. A. Liu, S. Uppal, G. S. Sukhatme, J. J. Lim, Englert, and Y. Lee. Distilling motion planner augmented policies into visual control policies for robot manipulation. Conference on Robot Learning, pages 641-650. PMLR, 2022.
[39] T. Jurgenson and A. Tamar. Harnessing reinforcement learning for neural motion planning. arXiv preprint 1oXiv:1906000/11111 2019.
[40] H. Ha, J. Xu, and S. Song. Learning decentralized multi-arm motion planner. arXiv preprint (:Xiv:20/1.02600 2020.
[41] P. Lillicrap, J. J. Hunt, A. Pritzel, Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1rXXiv:500.009999 2015.
[42] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprini urXiv:1707.066: 2017.
[43] R. Antonova, J. Yang, K. M. Jatavallabhula, and J. Bohg. Rethinking optimization with dif- ferentiable simulation from a global perspective. Conference on Robot Learning, pages 276-286. PMLR, 2023.
[44] E. Todorov, T. Erez, and Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012.
11

---

[45] R. Coulom. Efficient selectivity and backup operators in monte-cario tree search. In Interna- sional conference on computers and games, pages 72-83. Springer, 2006.
[46] Sola, J. Deray, and D. Atchuthan. A micro lie theory for state estimation in robotics. arXiv preprint arXiv: 2018.
[47] I.. Zaninetti and M. Ferraro. On the truncated pareto distribution with applications. Open Physics, 6(1):1-6, 2008.
[48] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[49] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl. Learning by cheating. In Conference Robot Learning, pages 66-75. PMLR, 2020.
[50] T. Miki, J. Lee, Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. In Science Robotics, volume 7.62, page eabk2822, American Association for the Advancement of Science, 2022.
[51] A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid Motor Adaptation for Legged Robots. arXiv preprint arXiv:2107.04034 2021.
[52] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning, pages 1722-1732. PMLR, 2023,
[53] M. Andrychowiez. F. Wolski, A. Ray, Schneider, R. Fong, Welinder, McGrew, J. To- bin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
12

---

A
Manipulation Planner implementation details
Implementation details and parameters for the manipulation planner are given this section.
A.1
Rewards
The three reward types of the manipulation planner are visualized in Fig. 5.
distunce reward
proximity reward
reachabiiity reward
low
robot
abject
high
Figure 5: Low and high reward states for the three rewards of the manipulation planner.
Reachability reward clipping The reachability measure m defined in (8) is clipped from below lower bound value mmmin E R4 in our implementation prevent overly large rewards, i.e.. m max(m, mmin), and the reward calculation (9) is updated to
I'm m
--m log
(12)
mmin
A.2 Node selection and extension horizon
Pareto distribution The continuous truncated Pareto distribution is defined
p(z; Tn, 3) Bx (8+11
(13)
na
for any E [1) and B determines the width of the distribution. The probability of sampling specific node index in is
ri+
1-8 (i+ 1)+
P(in = i; nu, B) =
p(r; Rn, 3) dr
(14)
n0
Varying search parameters Instead of using fixed values for the Pareto distribu- tion parameter 3 (greediness of the node selection) and the extension horizon (re- lated the task complexity), these val- modified during the search. The values updated depending whether the search progresses or not. The greed- iness of the node selection should be re- duced when stuck in a local minima, and the extension horizon can be interpreted
Algorithm 2 Node Extension Parameter Update
1:
in nee = node weteeeseeeeetooneeee ne))
D Alg. In.
false 4: for is in m n. do
3: better
B Alg. In.
5: a,
action sampling(p.) 8 n, = extension sn,,, (n,+1)
6:
7:
if is best boaesttnode(ss..) then
8:
3 = Buax
n1 0.95m, + 0.05 + +)
10: better node = true
as a measure of task complexity. An in- tuitive example of the meaning of the tension horizon is the Rubik's is cube, where one must execute sequence of actions to improve the cube's configuration. Here, some worse intermediate states are neces- sary to improve the overall configuration.
11: n. = nu
12: in =
13: if better node false then
14: 8 max (0.993, 5mmn)
15:
min (0.95n. + 0.05(ne + 1), ne,
16: nn n. +
Alg. 1, In. 10
17:
We bound 8 E [Bmin Smax] with lower and upper bound Bmins 3max R4, respectively, and bound no E [1, me,]ax] with upper bound ne,max N4. In the inner-most for-loop of Alg. I (In.
13

---

7), we check if the search progresses, i.e., if a new node with the overall highest reward is found. In this case, we update the parameters so the node selection is greedier and the extension horizon approaches the number extensions required for search progress. If no new best node is found, we update the parameters so that node selection less greedy and the extension horizon approaches the upper horizon bound. These updates Alg. are stated Alg. 2.
A.3 Actions
The four action types of the manipulation planner are visualized in Fig.
action
previous action
rundom
continuation
robot
object
proximity
goal direction
Figure 6: Visualization the four action types of the manipulation planner.
Varying action step size It can be helpful to have actions with different time step sizes to allow both short, precise motions and long, far-reaching motions. We randomly sample the action step size as an integer multiple of the base step size Ata, i.c..
A kåta, with k U(1, kmax),
(15)
where kimx E N is the upper bound for step size scaling. After the planner's search finished, we split all actions the base step size to obtain one common action step size for learning.
Closed-form solution for goal-directed action problem (11) is
The closed-form solution for the optimal control
Aag - BBQB+B+ (OBB+R)"BBBAA (BTQ sg) +Raj)
(16)
A.4 Tree extension
Dynamics rollout controller u -Kpp a,) Kaq.,, with controller gains Kp and Ka. A low-level control step size At. < At.. smaller than the action step size is used to generate a position reference
The actions of the planner are position commands tracked by low-level PD
At. tAte
tht
aref,t = (gr.j. a.
+ (qr,i, + a...)
(17)
At
At
The reference is linear interpolation between the last commanded position for the parent node, Ar.in + aj,, and the last actual position plus the newly computed relative action, qr.i. ai.. which creates a continuous control input. The same control scheme is used for RL policy actions.
A.5 Parameters
The planner's parameters that are not task-specific are listed in Table 2.
Parameter
Value
Parameter
Value
goal bias bg
0.0
Pareto parameter upper bound Buax
1.2
number of goals flg number of extensions ne reachability reward bound Muiin
30
Pareto parameter lower bound Buin
0.2
30
extension horizon bound Ne,aux
10
0.001
base action time step At.
0.4s
action time step scale bound kmax
3
Table 2: Planner parameters.
14

---

B
Reinforcement learning implementation details
Implementation details and parameters for reinforcement learning are given in this section.
B.1 Pure DDPG
Our reinforcement learning setup largely fol- lows the algorithm outlined in [53]. The pol- icy network consists of four hidden layers with 256 neurons each. We use ReL.U and tanh out- put activation functions ensure the actions lie within [-1, I]. The value function network also uses four hidden layers with 256 nodes each and ReLU activations but no output activation. We clip the value function's ss objective to the minimum value possible for the rollout horizon to avoid the unbounded divergence value es- timates. To improve stability during training, employ target networks with Polyak aver- aging updates for both the policy and the value function. A normalizer estimates the mean and standard deviation for each input and normal- izes all input values to zero mean and unit vari- ance. The exploration noise consists of actions chosen uniformly at random with probability of 17 and additive Gaussian noise for the actions chosen by the policy. complete list of hyperpa- rameters is given in Table 3, and the whole algorithm is stated in Alg. 3. We stop the training if the policy achieves success rate of 1.0 or if the maximum number of epochs has been reached.
Hyperparameter
Value
policy layers value function layers neurons per layer
4
256
policy learning rate value function learning rate
0.001
0.001
Polyak averaging factor random action chance
0.05
0.3
discount factor epochs Peepochs
0.98
task specific
cycles neycles
50
rollouts Ryollouts
2
rollout horizon fsstes
task specific 40
training episodes Repisode
training batch size Wbbtchh
256
replay buffer size evaluation runs per epoch
10000N
10
Table 3: Training hyperparameters.
Algorithm DDPG Algorithm
1: Initialize policy To and value function Q with random weights
2: Initialize targets Q' and n' with weights ø' Φ. 0'
3: Initialize replay buffer B
4: for epoch in 1 Repochs do
5: for cycle in 1 Peyclls do
6: for rollout in Prollouts do
7: Sample start state so and goal state Sg
8: for in Tasteps do
b policy rollout
9: if rand() < 17 then
Sample uniformly random action a; 11: else
Compute action a, (o, Sy) + Mg, with exploration noise Aç
13:
Take action and obtain new state 8t+1 Store transition (8t, at, 8g.t; Ft; 81+1) in
15:
Update normalizer with new samples for episode in 1 Repisode do Sample minibatch of Pbatch transitions (St. at, 8g.ti Tt. 8(+1) from B
normalizer update
network training
17:
Compute value target V; = r; + 7'' (s++1, Te (81+,, 8g (g.it)) (g.t)))
19:
Compute value function gradient 9Q for lg = Abatch E,(, - Qa ai 8),)))
Compute policy gradient 9x for L Apply the update T and Qe
Mbatch Σ, Qo(s, To sg.i), sg.i)
22: Update the target networks: ø' = TO + (1 _)' , 0' = -0 + (1-T)0
23:
Evaluate policy and stop if success rate 1.0
15

---

B.2 Demonstrations in replay buffer
Instead of generating all data with an on- line policy rollout, data from the plan- ner's search tree can be used with bias bp [0, I]. Accordingly, the rollout loop of Alg. (In. 6) is modified to use both the policy and the planner to generate tra- jectories, as stated in Alg. 4.
Algorithm 4 Planner Rollout
1:
2: for rollout in Prollouts do
D Alg. In.
3: if rand() I bp then
Sample goal state 8g
5: Find node closest to $g
Find path from root node to closest node
7: Clip pad trajectory if necessary
Planner demonstration lengths vary and can be shorter or longer than the rollout
Store all transitions in
9: else
horizon Mestops- For long trajectories, the
10: Sample so and 8y
P Alg. 3, In.
beginning is cut so that only the last Natees 11: for / in 1 N do
D Alg. 3, In. 8
transitions remain since these transitions are most likely to contain reward signals.
12:
Short trajectories are padded by repeating the planner's start state to reach a length of Natops- The action commands for the padded start states are the initial joint states the robot.
B.3 Pre-training
A policy network JØn of the same size as тa can be trained with imitation learn-
Algorithm 5 Pre-training
ing from planner demonstrations. The pol-
1: for demo in Addemo
> demo generation
2: Sample goal state 8g
icy trained with input (s, 8g) and out-
3: Find node closest to 8g
put (a) from the demonstrations. Simi-
4: Find path from root node to closest node
larly, the value function is trained with in- put (s, $g. a) and output (e), where the discounted value for input tuple can be calculated trivially from the rewards (I) for all states in a demonstration trajectory. Algorithm states the pre-training proce- dure. Note that the policy and the value function can be pre-trained separately.
5: Store all transitions in B if goal reached
6: Compute value for all transition
7: for episode in Repisode do network training
8: Sample minibatch of Thatch tuples (St, att 8g,t, U4) from B
9: Compute value function gradient gQ for le =
Thatch E( Qa(si, as 8g,))²²
10:
Compute policy gradient 9- for lx Eg1la - T $g,i)l2
The pre-trained policy and value function are used as described in [25]. In sum- mary, during the policy rollout and net- work training phase, the better one of the two policies, as evaluated by the value function, used. Note that we use the same value function for pre-training and online training.
11:
Apply the update to T and Qe
C Experiment details
The evaluation systems are described in this section. Afterward, several ablations and comparisons for the manipulation planner and reinforcement learning are provided.
C.1 Evaluation systems
Table 4 provides the task-specific parameters for the evaluation systems. Note that all object velocity states have scaling Qa,o.g = 0.17 and the robot state scaling zero: Qa,; 0I. A description of each system is provided below.
Box push The box push task is a one-dimensional pushing task where one actuated pusher box (robot) must push second unactuated box (object) to goal position on line. Due its simplicity, this task give valuable insight into the general properties of planning and learning. Moreover, only pushing but pulling is possible, which creates interesting manipulation challenge.
16

---

System
Nepochs
Msteps
Pa
Qd.o.g
4
qn
Box push
150
75
[1 2 2]
diag(1)
Box push 2D
150
100
[6
2 2 1]
diag(1,
0.0011
0.001
Planar hand
150
80
[1 [
diag(1, 1,
0.011
0.01
Floating hand
150
100
[1 [111]
diag(1, 2, x/2)
0.011
0.01
Allegro hand
300
50
[1322]
diag(100, 100, 100, 1, 10)
0.011
0.01
Box grasp
300
50
[0 [0111]
1,
0.011
0.01
Ball turn
300
75
[2 [2133]
0.11
0.1
Stool lift
300
75
[1111]
0.1
Table 4: Task parameters.
Box push 2D The two-dimensional box push task is the direct extension of the one-dimensional box push task. Just the one-dimensional box push task, it is illustrative example. The com- plexity of the task is increased by placing the pusher box between the object box and the goal. This setup requires the pusher to move around the obstacle before pushing it towards the goal.
Planar hand The planar hand is a simplified representation of in-hand manipulation a two- dimensional plane. A hand with two two-link fingers is used to rotate and lift a box. Even with the restriction plane, this task requires dexterous manipulation and gives insight into the planner's ability to solve in- in-hand manipulation tasks.
Floating hand The floating hand has the same kinematic as the planar hand and is also limited to a two-dimensional plane. However, instead of a fixed wrist, the floating hand can move in the plane. The task is to reorient a box and move it in space. This task combines dexterous manipulation as well as picking and placing an object, two commonly required skills.
Allegro hand The Allegro hand is robotic hand by Wonik Robotics with four fingers, sixteen actuated finger joints, and two actuated wrist joints. The task is to reorient cube by turning it around the upright z-axis. In-hand manipulation is commonly investigated robotic skill due to its high dimensionality and required dexterity.
Box grasp The box grasp task uses a stationary quadrupedal robot with a mounted arm to grasp and lift box with a handle. is a classic pick-and-place task, where finding proper grasp the main challenge. This task demonstrates method's ability to solve pick-and-place scenarios real systems.
Ball turn The ball turn task involves quadrupedal robot on its back that using its legs to reorient a ball. This setup resembles in in-hand manipulation tasks where multiple multi-joint fingers (legs) are used to reorient an object. We use this setup to demonstrate our method's ability to perform in-hand manipulation on real systems.
Stool lift The stool lift task comprises two stationary quadrupedal robots with mounted arms that lift a bar stool into an upright configuration. The robots are deliberately prevented from grasping the stool by fixing their grippers in closed state. In this manner, whole-body and bimanual manip- ulation of the stool is required, with contact occurring on multiple parts of the robot arms. We use this task as an exemplary real-world cleaning scenario, where robots can help put moved or fallen furniture back into desired configurations. By forcing whole-body contact, we also demonstrate our method's ability perform whole-body manipulation real systems.
C.2 Manipulation Planner
We present additional search results for two tasks and several studies the planner's components and parameters.
17

---

C.2.1
Additional tasks
Box push and box push 2D results Extending the results presented in Fig. 2, we show the search progress for the two additional tasks, box push and box push in 2D, The results are in line with the reported results for the other tasks.
Box push
Box push
1.0
scarch progress
0.5
0.5
0.0
0.0
0
2000
4000
6000
0
2000
4000
6000
simulation steps
simulation steps
Figure 7: The performance of the Manipulation Planner two additional tasks. Relative closeness to the goal for the best node in the tree so far vs. the total number of nodes (simulation steps).
C.2.2 Baseline comparison
A comparison of three manipulation tasks with increasing difficulty based on the planar hand is given in Fig. 8. The tasks are (1) in-hand reorientation of a box by 90", in-hand reorientation of box by 180%, and throwing the box onto a shelf. The 90 turn can be achieved by simply tipping over the box. For 180", regrasping is required. Throwing the box onto the shelf requires highly dynamic movements to overcome the gap between the hand and the shelf. We compare our planner the quasi-static planner [14].
Planar hand 90"
Planar hand 180
Planar hand throw
1.0
1.0
1.0
progress
0.5
ours quasi-static
0.0
0.0
0.0
2000
4000
6000
2000
4000
6000
0
2000 4000
6000
simulation steps
simulation steps
simulation steps
Figure 8: Performance comparison of our planner with the ofate-of-thh-art planner introduced in [14] for using the planar hand reorient a box 90", 180", and throw the box into desired location. Relative closeness to the goal for the best node in the tree so far vs. the total number of nodes (simulation steps).
We outperform the quasi-static planner across all three tasks. It is important to note that, for the throwing task, the search progress does not fully capture performance. While our planner consis- tently succeeds in throwing the box onto the shelf, the quasi-static planner fails to find this solution. These results underscore the necessity of dynamic planning for advanced manipulation tasks and highlight our planner's superior capability solve such tasks, thereby outperforming state-of-the- art planner.
C.2.3 Ablations
Extension horizon The extension horizon specifies how many sequential actions are taken once a node has been selected to extend the tree. We assume that tasks different complexity require different extension horizons. We evaluate different but fixed extension horizons for all tasks to measure the influence on the planner's search progress in Fig. We also evaluate the automatically varying horizon that use in our implementation as described in Appendix A.2. We measure the average closeness to the goal over the full search duration, which is larger for quicker search progress and higher final closeness. While a longer extension horizon tends to improve performance
18

---

overall, there is no strong correlation. At the same time, the varying horizon performs well in all cases with reduced variance compared to the fixed horizons.
Box push
Box push 2D
avernge progress
1.0
fixed horizon
0.5
0.5
variable horizon (ours)
0.0
0.0
10
Planar hand
Floating hand
Allegro hand
average progress
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
10
Box grasp
Ball turn
Stool
average progress
1.0
1.0
1.0
0.5
0.5
0.0
0.0
0.0
5 extension horizon
10
5
10
10
extension horizon
exlension
Figure 9: The performance of the Manipulation Planner for different extension horizons measured as the average closeness to the goal during the search. Comparison of different fixed horizons (blue) and variable horizon (black).
Pareto distribution exponent The Pareto distribution parameter 3 modifies how greedily nodes with high rewards are extended. We evaluate different but fixed Pareto parameters for all tasks to investigate how the parameter influences the planner's search progress in Fig. 10. We also evaluate
Box push
Box push 2D
average progress
1.0
1.0
fixed parameter
0.5
0.5
variable parameter (ours)
0.0
0.0
2
Planar hand
Floating hand
Allegro hand
average progress
1.0
1.0
1.0
0.5
0.5
0.0
0.0
0.0
Box grasp
turn
Stool
average progress
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
2
0
Pareto parameter
Pareto parameter
Parcto parameter
Figure 10: The performance of the Manipulation Planner for different Pareto parameters measured as the average closeness to the goal during the search. Comparison of different fixed parameters (blue) and variable parameter (black).
19

---

the automatically varying parameter that we use in our implementation as described in Appendix A.2. For intuition, a parameter 3 = 0.05 means that the currently best node will be selected with a probability of roughly = 10,, whereas 3 = 2.0 means a probability of P = 75%. There a consistent trend that the planner's performance decreases as the node selection becomes more greedy (the parameter increases). This effect could be caused by the node selection becoming so greedy that the search gets stuck in local minima by repeatedly selecting the same few nodes. Using an automatically varying range for the parameter does not lead to improvement over always choosing a low parameter. Apparently, the Pareto distribution with a low B already has a good tradcoff' between exploitation and exploration.
Sub-goal ratio To assess the usefulness of having sub-goals during the planner's 's search, as in rapidly exploring random trees, we evaluate different ratios of sub-goals to node extensions in Fig. 11. A goals/extensions ratio of 1/900 means we choose one goal and do 900 extensions for this single goal, whereas a ratio of 900/1 means we choose 900 goals and do one extension for cach of these goals. Note that in this evaluation, we always do total of 900 extensions. There is no consistent trend indicating that more less sub-goals improve search performance. Multiple explanations exist for these results: Our tasks could have very few local minima. However, since the Pareto parameter does influence search performance (ef. Fig. 10), there may be many local minima, but our node selection strategy is already good enough to avoid these. Yet another possibility is that the difference in sub-goals is too small to provide a benefit over searching with just a single goal.
Box push
Box push 2D
average progress
1.0
1.0
fixed ratio
0.5
0.5
fixed ratio (ours)
0.0
0.0
10
-3
10
10a
10-3
10P
10
Planar hand
Floating hand
Allegro hand
uverage progress
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
10-3
3
100
103
10-3
100
10a
10
30
100
103
Box grasp
Ball turn
Stool lifi
average progress
1.0
1.0
0.5
0.5
0.5
0.0
10-3
100
103
10-3
100
103
10
100
103
goals/ extensions
goals / extensions
goals/ extensions
Figure 11: The performance of the Manipulation Planner for different ratios of goalsfextensions measured as the average closeness to the goal during the search.
Action types We investigate the effectiveness of different combinations of action types with just random actions as a baseline in Fig. 12. The fact that purely random actions already lead to good results in many tasks indicates that the other components of the planner, such as the heuristic node selection and specific rewards, lead to meaningful search progress. Still, proximity and goal-directed actions alongside random actions in the search further improve the performance for some tasks.
Proximity and reachability reward scaling The effect of different proximity and reachability reward scalings is shown in Fig. 13. For most tasks, the search fails once the proximity reward becomes too high, potentially because the necessary distance between the robot and the object for
20

---

Box push
Box push 2D
gg
rp
Planar hand
Floating hand
Allegro
Box grasp
Ball rurn
Stool lift
The performance of the Manipulation Planner for different combinations of action types measured as the average closeness to the goal during the search. Comparison of the following action types (blue): goal-directed (g), random+goal- direecee (rg), aandom-poximmty--goal directed (rpg), random-proximity (rp), proximity (p), proximity+-oall-dieette (pg), and random (black).
Figure 12:
reconfiguration is punished too severely. The reachability reward scaling shows similar, although weaker, trend. Especially for the more complex tasks involving the Allegro hand or robot arms, it difficult to clearly state how to scale the rewards, and parameter sweep is necessary to find the optimal value. It can be assumed that the same effect would occur for dense-reward reinforcement learning, highlighting one of the benefits of the planner, as parameter sweeps can be executed much faster with the planner than with learning.
Box push
Box push 2D
prosimity reward
10
10a
10-2
0.5
10-5
10
10-2
10a
10
10-2
10
Planar hand
Floating hand
Allegro hand
proximity reward
10
10a
10
2
10-2 2
10-3
10-5
10-5
10
10-2
10
10
10-2
10
10-5
10-2 10
grasp
Ball turn
Stool lift
proximity reward
10a
10a
10a
10-2
10-2
10-2
10-5
10
10-6
2
101
10-6
10 2
10"
5
10-2
10a
reuchability reward
reachability reward
reachability reward
Figure 13: The performance of the Manipulation Planner for different combinations of action types measured as the average closeness to the goal during the search. Comparison of different proximity and reachability reward scalings. Purple is bad, yellow good.
21

---

C.3
Reinforcement learning
We present additional learning results for two tasks and several comparisons for different learning methods and demonstration usage.
C.3.1 Additional tasks
Box push and box push 2D results Extending the results presented in Fig. 3, we show the training progress for the two additional tasks, box push and box push in 2D, The results are in line with the reported results for the other tasks.
Box push
Box push 2d
1.0
1.0
success Tale
pure RL
with pre-training
RL with demos (ours)
0.0
0.0
x 10"
0
1 x 10" simulation steps
simulation steps
The performance of reinforcement learning for manipulation on two additional tasks. Comparison of our approach of adding demonstrations to the replay buffer (blue), RL with an addi- tional imitation policy pre-training (red), and pure RI. (black).
Figure 14:
C.3.2 Planner and policy comparison
While the planner is able to solve the manipulation tasks presented in this paper, distilling policy from planner demonstrations leads to more optimal and faster solutions (after training).
Solution length comparison The trajectory lengths (in steps) for planning and learning are given in Tab. 5. We list the average trajectory lengths for five seeds.
Task
Planner
Policy
Box push
80.4
59.4
Box push 2D
110.6
37.6
The results show that both the planner and the trained policies are able to find solutions. However, the policies always require significantly fewer steps to solve the tasks, indicating a more optimized performance.
Planar hand
72.6
17.0
Floating hand
81.2
47.8
Allegro hand
41.0
30.4
Box grasp
43.4
21.2
Ball turn
34.0
21.2
Stool lift
57.8
28.4
Timing comparison The wall-clock times for planning and learning, as well as dimensions for the manipulation tasks, are given in Tab. 6. We list the average time for five seeds. Single sol. refers to finding single trajectory leading to one goal. Multiple sol. refers finding mul- tiple solutions leading to different goal locations in order generate of demonstrations for policy learning. Policy training refers to training the policy where demonstrations replay buffer. The planner is no longer used at this stage. Observation dim. refers to the combined dimensions of state and goal space. Action dim. refers to the dimension of the action space.
Table Solution trajectory lengths (in steps) of planner and trained policies for different manipulation tasks.
fed to the
The results show that both planning and policy learning scales high-dimensional tasks with only moderate increase in computation time.
C.3.3 Baseline comparison
We provide comparison to the state-of-te sate-of-the-ann manipulation learning algorithm MoPA-RL [37] for the same three planar hand tasks described in Appendix C.2.2 and for complex assembly task.
22

---

Task
Single sol.
Multiple sol.
Policy training
Observation dim.
Action dim.
Box push
3.1s
15.3s
19.6min
8
Box push 2D
7.6s
13.9s
19.5min
16
Planar hand
6.7s
15.9s
5.9min
28
Floating hand
8.6s
17.5s
16.4min
36
6
Allegro hand
3.8s
43.7s
67. 67.0min
98
18
Box grasp
7.7s
40.0s
68.9min
54
Ball turn
4.
57.6s
58.1 1ii
74
12
Stool lift.
8.5s
56.2s
46.4min
84
14
Table 6: Wall clock times and dimensions for planning and policy learning of different manipulation tasks.
Planar hand We sparse rewards for the baseline comparison the planar-ham hand tasks and show the results in Fig. 15.
Plunar hand 90"
Planar hand 180
Planar hand throw
1.0
1.0
1.0
ours
MoPA RL
success rate
0.5
0.5
0.0
0.0
0.0
2 x 10
x 10"
0
2 x 10" simulation steps
4 x 10"
0
x 10
4 x 10"
simulation steps
simulation steps
Figure 15: Performance comparison of our full algorithm with MoPA-RL [37] for using the planar hand to turn box 90", 180", and throw the box into desired location.
We outperform MoPA-RI. for all three tasks. While the MoPA-RI. is also able to quickly and consis- tently find solutions to the 90% v0'-reorientation task, it has more difficulty with the 180" reorientation. Finally, MoPA-RL- is not able to find any solution to the throwing task, whereas our method achieves a success rate of more than 50%. As before, the results demonstrate the need for dynamic planning and our planner's 's ability to solve highly dynamic tasks.
Sawyer assembly With the Sawyer assembly task described in [37], we demonstrate our ability to solve complex assembly tasks that require precise manipulation in cluttered environments. The task challenging because colliding with the movable table prevents assembly. Accordingly, collisions must be avoided. We evaluate five seeds for our method, and we reuse the results shown in [37] for MoPA-RL.
Sawyer planning
Sawyer learning
1.0
1.0
search progress
Successs mite
ours MoPA-RL
0.5
0.5
0.0
0.0
0
2000
4000
6000
1.5 10"
simulation steps
simulation steps
Figure 16: Left: The performance of our planner for solving the Sawyer assembly task presented in [37]. Relative closeness to the goal for the best node in the tree so far vs. the total number of nodes (simulation steps). Right: Performance comparison of our full algorithm with MoPA-RL [37] for the Sawyer assembly task. Results for MoPA-RI. are taken from [37].
In the majority of cases, our planner is able find solutions for this assembly task. Policy learning successful in these cases with good planner demonstrations, While we achieve a similar average performance for policy learning MoPA-RL, we have higher variance due the reliance on the planner's demonstrations. While our planner is designed for contact-rich manipulation, it can also
23

---

solve this assembly task where contacts must be avoided. Our method implicitly takes collision avoidance into account and matches the performance of MoPA-RI.
C.3.4 Ablations
Demonstration ratio Adding demonstrations the RL replay buffer improves performance com- pared to pure online learning, but pure offline learning with the demonstration data does not yield good results. Accordingly, there must be tradeoff between online exploration and offline data. We investigate this tradeoff by varying the ratio of demonstrations in the replay buffer in Fig. 17. We measure the average reward over the entire training duration, which is larger for quicker learning progress and a higher final success rate. appears that small amount of demonstrations gives enough reward signals to bootstrap RL and enable exploration from sub-optimal solution. Already having low ratio of roughly 10% yields satisfactory results, and 25% it gives the most robust performance for all tasks.
Box push
Box push 2D
1.0
1.0
demonstration ratio
average
demonstration ratio (ours)
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Planar hand
Floating hand
Allegro hand
average success
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Box grasp
Ball turn
Stool lift
average success
1.0
0.5
0.5
0.5
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
demonstration ratio
demonstration ratio
demonstration ratio
Figure 17: The performance of reinforcement learning (RL) for manipulation measured as the average success during training eight tasks. Comparison of different demonstration-to -onlinnee- exploration ratios in the replay buffer. A ratio of 0.0 is pure online RL, 1.0 is pure offline RL.
Demonstration inclusion comparison Three different methods for adding demonstrations to replay buffer are investigated. (1) Use fixed ratio of demonstrations online exploration. (2) Use variable ratio, where the ratio decreases the learning success rate increases. (3) Add fixed number of demonstrations from the planner's 's tree only initially. As can be seen in Fig. 18, the spe- cific method of introducing demonstrations into the replay buffer does not make large difference. However, just adding demonstrations in the beginming is sometimes not enough to bootstrap learning successfully.
Pre-training comparison Directly using the pre- pre-trained policy only works for the most simple box push task with success rate of 0.10 0.13. For all other tasks, the success rate was zero. We evaluated runs for each task. Apparently, the data generated by the planner not good enough to directly perform imitation learning with it, and online exploration with RL is required to find successful policies.
24

---

Box push
Box push 2D
1.0
1.0
SUCCess rate
RL initial demos
0.5
0.5
RL w. decreasing demos
RT. constant demos (ours)
0.0
0.0
x 10
0
1 x 10"
Planar hand
Floating hand
Allegro hand
1.0
1.0
1.0
SUcOoss rate
0.5
0.5
0.5
0.0
0.0
1 x 10
0
x 10
1 10
Box grasp
Ball turn
Stool lift
1.0
1.0
1.0
success rate
0.5
0.5
0.5
0.0
0.0
0.0
10 simulation steps
10
x 10"
1 x 10
10"
simulation steps
simulation steps
Figure 18: The performance of reinforcement learning for manipulation with demonstrations on eight tasks. Comparison of adding constant ratio of demonstrations the replay buffer (blue), adding fewer demonstrations with increasing success rate (dark blue), and adding demonstrations only at the beginning (purple).
Accordingly, we compare three different pre-training settings in Fig. 19. (1) Only pre-training a policy TIL and using alongside the online RL policy. (2) Pre-training a policy and the value function. (3) Pre-training the policy and value function as well as adding a fixed ratio of demonstra- tions to the replay buffer. While adding a pre-trained policy RL improves performance over pure DDPG, it does not significantly and consistently perform much better. Only once demonstrations are added to the replay buffer does the performance with a pre-trained policy improve significantly. Since the performance of adding demonstrations alongside pre- tre-training is not better than just adding demonstrations, we attribute this improvement to using demonstrations and not to the pre-trained policy.
HER comparison For reference, we provide comparison to hindsight experience replay (HER) in RL since this method can improve learning performance in sparse-reward settings. The results shown in Fig. 20. While using HER improves the learning performance for some tasks compared to pure DDPG, it does not lead to any improvement for others and almost always performs worse than adding demonstrations to the replay buffer.
C.4 Hardware evaluation
Setup We use the quadrupedal Spot robot by Boston Dynamics with an arm arm mounted for certain tasks. Control and communication with the robot are achieved via the bosdyn software de- velopment kit. Accurate state estimation achieved by using the OptiTrack motion capture system and placing markers the objects. Communication between the Optitrack System and the robots was handled via ROS2, while gRPC was used to command policy actions with the Spot robots.
25

---

Box push
Box push 2D
1.0
1.0
success rate
RL policy pre-training
0.5
0.5
RL w. full pre-training
RI. w. pre-training & demos
0.0
0.0
1 x 10
0
1 10"
Planar hand
Floating hand
Allegro hand
1.0
1.0
1.0
SUcOOss rate
0.5
0.5
0.5
0.0
0.0
x 10
0
x 10"
1 X 10
Box grasp
Ball turn
Stool lift
1.0
1.0
1.0
success rate
0.5
0.5
0.5
0.0
0.0
0.0
10 simulation steps
x 10
x 10"
1 10
10"
simulation steps
simulation steps
Figure 19: The performance of reinforcement learning for manipulation with pre-training on eight tasks. Comparison of pre-training just the policy (light red), pre-training the policy and value func- tion (red), and pre-traie pre-training the policy and value function as well as adding demonstrations (dark red).
Box push
Box push 2D
1.0
1.0
SUCCOSS rate
pure
RL with HER 50%
0.5
0.5
RL. with HER 80%
RL with demos (ours)
0.0
0.0
1 x 10
0
1 x 10" Floating hand
Planar hand
Allegro hand
1.0
1.0
1.0
SUcooss rafe
0.5
0.5
0.5
0.0
0.0
1 10
0
1 10 6
1 1 10
Box grasp
Ball turn
Stool lift
1.0
1.0
1.0
suceoss rate
0.5
0.5
0.5
0.0
0.0
0.0
0
1 10 simulation steps
x 10
x 10
1
10
10
simulation steps
simulation steps
Figure 20: The performance of reinforcement learning for manipulation with hindsight experience replay on eight tasks. Comparison of our approach of adding demonstrations to the replay buffer (blue), RL with 50% HER relabeling (light green), RL with 80% HER relabeling (dark green), and pure RL. (black).
26

---

Box grasp The box grasp task evaluates if the robot is able to grasp and lift the box. A trial is considered successful the box is lifted off the ground. We evaluate 27 different starting locations by building a grid of start positions and orientations for the box:
x E { 0-0.1 0.13 x E f 0.25, 0.00, 0.25) x 02 E (-455, ,0,55500,0440 4.
(18)
Fig. 21 shows the progression of the box-grasp task. The challenge in this task is to learn that in order to lift the box, the handle must be grasped. Our trained policy is able to consistently grasp the handle and performs well.
Initial box configuration
0.0s
4.8s
9.
9.85
Figure 21: Left: the successful and failed initial configuration for the box-grasp tasks. Right: deployment of the trained policy for one Spot grasping and lifting the box.
Ball turn The ball-turn task is used as an in-hand menhaand-maiipe cotnipulaaton-equivvan considered successful if the box rotated such that the angle deviation from the upright z-axis is below 40", We evaluate 20 different starting locations by testing sequence of start rotations around the y-axis for the ball with increments of 15
example. trial is
0g 1800 1165 ,... ,., ,460),
(19)
Fig. 22 shows the progression of the ball-turn task. The robot successfully perform some of the desired orientations. We attribute failures the sim2real gap, with rigid ball in simulation training and soft yoga ball in the real-system setting.
initial ball angle
2S.8S BS
25, 85
dtgree))
Figure 22: Left: the successful and failed initial orientations for the ball- aall-turn tasks. Right: deploy- ment of the trained policy for one Spot turning the ball to the upright position.
Stool lift The stool-lift task requires whole-body manipulation and is bimanual task representa- tive of a rorl-world cleaning task. A trial is considered successful if the robots maneuver the stool in the upright position. We evaluate 25 different starting locations by building grid of start positions for the stool:
E 0.10022-01110000,0111.. x (-0.2, - 0.1, 0.0, 0.1,
(20)
Fig. 4 shows the progression of the stool-lift task. Despite the challenging whole-body and bimanual setup, the robots frequently manage to lift the stool in the upright configuration. Failures can be attributed to lack of measuring the robots' body positions. When contact forces occur, the robots' bodies are displaced (which we do not measure), leading to position offsets for the robot arms. In such cases, the policy has false state measurements and produces unsuccessful actions.
27


Reviews:
[{"review_id": "qHlx8HloeB", "reviewer": "Reviewer_tB8p", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The motivation of this paper is that collecting human demonstration data is costly and not scalable. One would like to use motion planners to generate some demonstration data. However, the motion planner often runs under the assumption of quasi-static dynamics, and the resulting demonstration data might lack action labels. To address this challenge, the paper proposed to use the gradient descent method that computes the gradient through the planner. The resulting generated data is used to train control policies. The goal is not to develop a method that improves the solution-finding capability of the planner but to use that dataset to bootstrap the learning policy.  To find the solution given the dynamics or simulation of the robot, the paper proposes to use tree search methods to find a good planner solution for the whole-body manipulation. The process goes through (1) goal sampling, (2) reward update, (3) node selection, (4) action selection, and (5) transit to the next state given action and the current state.   The second half of the paper talks about how to use such data to train RL policies. The first approach is to add data to the replay buffer, and the second approach is to pre-train the Q function on such an offline dataset.  Finally, the proposed approach is tested in real-world robot tasks.", "summary_of_recommendation": "For the above weakness, the paper is on the borderline.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Could authors give an example of how fast the proposed method can run (wall clock time) under what dimension of the states and actions?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vVNWoaMzha", "reviewer": "Reviewer_feZG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "the paper propose a planner for boostrapping RL for manipulation tasks", "summary_of_recommendation": "strong real-world experiments and the technical details seem sound", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Can you compare to other previous state of art algorithms? Or some way to show improvement of your method beyond existing state of art algorithms? This seems possible in sim, potentially.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IJZhX9Srnn", "reviewer": "Reviewer_YCCA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a flexible and general manipulation planner that can generate dynamically feasible trajectories for dexterous and whole body manipulation. The planner is used to generate demonstrations for guiding a reinforcement learning agent in simulation. The learned manipulation policy is then deployed zero-shot on a real robot system. The overall pipeline is evaluated on manipulation tasks showing both efficient learning and reasonable sim-to-real transfer.", "summary_of_recommendation": "The paper makes two contributions: development of a model-based manipulation planner and using the planner to guide an RL agent. I am unable to recommend acceptance because I am not convinced of the novelty and significance of the planner due to a lack of baselines.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Please experimentally compare the proposed planner with baselines, even if they make the quasi-static or quasi-dynamic assumption. For example, the authors have cited [14] and [15].\n1. Please clarify how the planner compares with and differs from [1] and [2] (see review). Ideally I would like to see an experimental comparison but a discussion may suffice if the authors can clearly articulate how their method is superior.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFFpOcYSY1", "reviewer": "Reviewer_RBak", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a two-stage framework for learning manipulation policies, wherein the first stage, motion-planning is used to generate expert demonstrations removing the need for collecting human-generated expert trajectories. In stage 2, these demonstrations are used to augment the replay buffer of an off-policy deep reinforcement learning for bootstrapping the learning process for various dexterous and whole-body manipulation tasks.", "summary_of_recommendation": "The paper is clearly written and easy to follow. While the problem formulation as well as the experiments are convincing, there are several open questions that, if addressed, will help the community to further take this approach to a generalizable and scalable solution.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "I request the authors to address the concerns below:\n1. As acknowledged by the authors, the dependency of both the planner as well as the policy on the state space creates a bottleneck for this approach to scale to diverse tasks in the real-world. Another related work [1] further distills the state-based policy into an asymmetric policy which still relies on the state space for the critic, but distills the actor into a vision-based policy, which is more readily deployable in the real-world outside of constrained environments. Can a similar technique be used here for learning visual policies? Moreover, [1] and [2] seem to be highly relevant to this work and must be included in the related works.\n\n2. Motion-planning is classically beneficial for collision-free planning. Have the authors tried any tasks where motion-planning provides any additional benefits such as finding collision-free trajectories in cluttered environments? \n\n3. How can this method be extended to complex manipulation tasks which are non-trivial for motion-planning to solve, such as complex assembly tasks or dynamic object manipulation?\n\n4. While there are several benefits to using classical motion planners, what are the challenges and bottlenecks of using them in terms of both computational complexity as well as real-time efficacy? It would be helpful to know how much time and resources are allocated to collecting expert demonstrations for each of these tasks.\n\n5. It is unclear how the expert trajectories are collected for the collaborative task of stool-lifting with two arms. An elaborate explanation of that would be helpful.\n\n[1] Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation. \n\n[2] Motion Planner Augmented Reinforcement Learning for Obstructed Environments.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths: - This work integrates motion planners with RL to bootstrap learning and create an efficient learning approach that can be used in diverse applications like in-hand manipulation and lifting with robots. - Sim2real transfer of learned policies Weaknesses: - Lack of baselines: Authors must provide comparisons against baselines from literature including some mentioned by reviewers. How does the performance differ from a pure motion planning approach? Reviewers also raise concerns about the overall novelty of the proposed approach which could be addressed through writing and baseline comparisons too. - The heuristics used to get good demonstrations from the planner make the generalizability of the approach concerning. Authors should provide more motivation for why this is scalable to more tasks especially tasks that might be challenging for motion planning? - Dependence on state space in the real-world is constraining. How does performance deteriorate when using visual policies? 

Post-rebuttal discussion: The key points addressed in the rebuttal are: - Baseline comparisons and novelty: Authors have added comparisons with other approaches from literature and added a new dynamic throwing task in simulation. They have also added discussion about their approach versus others from literature aiming to better highlight their novelty. However the additional simulation comparison experiments are too simplistic and the paper could benefit from stronger baseline comparisons especially in the real-world, which we encourage the authors to include in the camera-ready version.
Decision: Accept