Jacta: A Versatile Planner for Learning Dexterous and Whole-body Manipulation

Authors: jan  br üdigam
ali-adeeb  abbas
maks  sorokin
kuan  fang
brandon  hung
maya  guru
stefan  sosnowski
jiuguang  wang
sandra  hirche
simon  le cleac'h
t u munich
boston  dynamics
a i institute
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 22:51:02

Abstract: Robotic manipulation is challenging due to discontinuous dynamics, as well as high-dimensional state and action spaces. Data-driven approaches that succeed in manipulation tasks require large amounts of data and expert demonstrations, typically from humans. Existing planners are restricted to specific systems and often depend on specialized algorithms for using demonstrations. Therefore, we introduce a flexible motion planner tailored to dexterous and whole-body manipulation tasks. Our planner creates readily usable demonstrations for reinforcement learning algorithms, eliminating the need for additional training pipeline complexities. With this approach, we can efficiently learn policies for complex manipulation tasks, where traditional reinforcement learning alone only makes little progress. Furthermore, we demonstrate that learned policies are transferable to real robotic systems for solving complex dexterous manipulation tasks.

Project website: https://jacta-manipulation.github.io/

Main paper: 
 introduction: 
 Dexterous and whole-body robotic manipulation in high-dimensional state-action space with nonsmooth dynamics is challenging, but learning-based approaches have shown encouraging results for such tasks. However, they require loads of high-quality data [1, 2] or extensive exploration to find solutions. Yet, robot data is scarce and expensive; human demonstrations are costly and confined to human capabilities. Moreover, using dense rewards or curriculum learning to guide the policy search potentially imposes a sub-optimal solution structure and necessitates lengthy reward shaping.A promising solution to these issues is to leverage motion planners to automatically generate demonstrations for learning instead of relying on humans [3, 4, 5] . This approach addresses the challenge of scaling data collection. However, existing manipulation planners are often designed for specific end effectors and single tasks such as pushing [6, 7] , pick-and-place, [8, 9, 10] or in-hand manipulation [11, 12] . Further, using the demonstrations generated by these motion planners is non-trivial because they assume quasi-static dynamics [13, 14, 15] or only explicitly provide state trajectories [12] . This paper shows that robotic manipulation must not be limited in this manner. A planner can combine highly dexterous capabilities to precisely manipulate small items and whole-body interactions to maneuver large objects, with contact occurring not just at the end effector.To address these issues, we introduce a novel planner and learning pipeline for dexterous and wholebody robotic manipulation, shown in Fig. 1 . The planner finds solutions to manipulation tasks through sampling-based and gradient-based actions for global and goal-directed exploration. It is not limited to quasi-static systems, nor does it need to explicitly reason about the contact modes. The aim is not to use the planner directly in a receding horizon fashion akin to model predictive control but to leverage planner-generated demonstrations to bootstrap policy learning. Thus, when using the planner, one is not bound to human demonstrations and human-like robotic systems. Instead, one can build on learning methods that are bootstrapped with the planner's solutions. Surprisingly, we find that simply adding a small number of planner demonstrations to the replay buffer is enough to drastically improve learning performance and generate successful policies for complex tasks.In summary, our contributions are the following:• Design of a novel motion planner generating dynamically feasible state-action trajectories for dexterous and whole-body manipulation tasks. • Identification of a simple yet efficient learning pipeline for using the planner's demonstrations in reinforcement learning. • Transfer of trained policies to real systems to demonstrate the feasibility of complex tasks covering both dexterous and whole-body manipulation. 
related work: 
 Guiding Manipulation Learning with Demonstrations Data-driven approaches are state-of-theart for solving complex manipulation tasks. Imitation learning (IL) leverages expert demonstrations of such tasks to train policies for robots. Recent results show the great potential of IL in advanced tasks, such as unseen object manipulation [16, 17] , door opening [18] , or manipulation of liquids and fabrics [19] . Yet, IL requires large amounts of expert demonstrations to deliver working policies, which is a bottleneck for quickly learning new skills, especially since human demonstrations are inherently limited to human capabilities and costly to collect. Reinforcement learning (RL) does not necessarily require demonstrations due to its noisy exploration. However, the initially random exploration in large state-action spaces has a low chance of success. Accordingly, demonstrations have also been used to guide RL with additional offline data. Two different ways of incorporating demonstrations into RL have found popularity: directly adding the demonstrations to the replay buffer with minor modifications to the underlying learning algorithm [20, 21, 22] or pre-training an imitation policy from the demonstrations [23, 24, 25] . Still, the advanced manipulation tasks investigated in [23, 24] only cover anthropomorphic systems and require human demonstrations. In contrast, we propose a pipeline with a planner that is not limited to anthropomorphic systems and eliminates the need for costly human data collection, thus offering greater scalability.Planning for Robotic Manipulation Learning While robotic motion planning has seen great success, especially with sampling-based methods [26, 27] , manipulation planning remains particularly difficult due to the multitude of contacts in general settings. Thus, most methods focus on motion planning for low-dexterity grippers and specialized tasks such as pushing or fixed grasping [6, 7, 8, 28, 29] . More advanced techniques plan dexterous manipulation assuming quasi-static contact dynamics [15, 14] and myopic sampling-based planning offers compelling results but only on short-horizon tasks [30] . Manipulation planning based on optimal control often requires smoothed contact dynamics [31] or scheduled contact sequences [32, 33] since contact-implicit schemes are limited to simple tasks in practice [34, 35, 36] . In contrast, our proposed planner is designed to solve dexterous and whole-body robotic manipulation tasks without restrictions to specific end effectors or contact locations. We allow contact on all parts of a robot and plan with full contact dynamics to generate readily usable demonstrations for manipulation learning.Since manipulation planners are not fast enough to be directly deployed on the robot systems, they have been combined with learning methods. Collision-free motion planners have been used as an alternative to RL's random exploration, but the applications have been limited to simple manipulation tasks [4, 3] . Moreover, motion-planner actions do not overcome the lack of reward signals in sparse reward settings [37, 38] . Demonstrations from motion planners have served as expert data for IL with pick-and-place tasks [5] and for RL with basic reaching tasks rather than full-scale manipulation [39, 40] . Planner-based demonstrations for manipulation have been used to find an informed start-state distribution for the RL exploration for simple object rearrangement tasks [13] , and a similar approach has been developed for dexterous in-hand manipulation [12] . However, these methods implement task-specialized planners and are not directly portable to generic manipulation settings. Additionally, they only provide state trajectories without associated actions, making it difficult to use the demonstrations directly for RL. Both of these issues are addressed with our planner, which produces dynamically feasible state-action demonstrations for general robotic manipulation tasks.Pareto distribution The continuous truncated Pareto distribution is defined asEQUATIONfor any x ∈ [1, n n ], and β determines the width of the distribution. The probability of sampling a specific node index i n = i isEQUATION)Algorithm 2 Node Extension Parameter Update we check if the search progresses, i.e., if a new node with the overall highest reward is found. In this case, we update the parameters so the node selection is greedier and the extension horizon approaches the number of extensions required for search progress. If no new best node is found, we update the parameters so that the node selection is less greedy and the extension horizon approaches the upper horizon bound. These updates to Alg. 1 are stated in Alg. 2. 
background: 
 This section describes the robotic manipulation scenario solved by our motion planner and the reinforcement learning algorithm used to optimize policies for the manipulation tasks.We consider one or multiple robots with joint configuration q r ∈ R nr and joint velocity1 qr ∈ R nr that interact with one or multiple objects with configuration q o ∈ R no and velocity qo ∈ R no . These quantities form the state s = [q T r qT r q T o qT o ] T ∈ R ns . The actions a ∈ R nr are position commands for the robots' actuated joints. The position commands a are tracked by a low-level controller (cf. Appendix A). The potentially non-smooth dynamics for this system are s t+1 = f (s t , a t ) at time t and contain bounds on the robots' and objects' states. The planner's task is to find a state-action trajectory that minimizes the weighted distance d(s, s g ) = ∥s -s g ∥ Q d to a desired goal state s g . The weight matrix Q d ∈ R ns×ns is used to prioritize certain dimensions of the state, e.g., of the objects, and ignore others, e.g., of the robots.Reinforcement learning algorithm Manipulation policies are found with value-based reinforcement learning [41, 42] . A goal-oriented value function Q(s, a, s g ) ∈ R for the system's stateaction-goal space is learned from rewards r. The sparse reward function r(•) ∈ {-1, 0} is 0 if the relative distance d of a state s t to the start state s 0 is below a threshold ϵ, and -1 otherwise:EQUATION)The optimal value function given this reward function isEQUATIONwhere γ ∈ [0, 1) is a discount factor. Approximating the value function with a network Q ϕ enables learning the network's parameters ϕ from tuples (s t , a t , s g , r t , s t+1 ) in continuous spaces. The optimal policy a = π ⋆ (s, s g ) maximizes the value function:EQUATIONand is also approximated by a network π θ with parameters θ, which provides a continuous policy representation. Since we want to use demonstrations from our motion planner for learning, we use the off-policy learning algorithm deep deterministic policy gradients (DDPG) [41] .4 Dexterous and whole-body manipulation planner for i e in 1 : n e do n n = n n + 1 11: i n = n nWe design a motion planner to generate dynamically feasible state-action trajectories for dexterous and whole-body manipulation. These solutions serve as demonstrations to bootstrap policy learning. The planner builds a tree starting from a single start state by taking actions towards the goal. For each action taken, a new node is added to the tree, as shown in Fig. 1 . In this way, the search space is explored until a viable trajectory to the goal is found.Robotic manipulation has discontinuous contact dynamics and rugged optimization landscapes [43] . Thus, the planner involves several components. Gradient-based actions for locally optimal search are combined with random actions for global exploration and avoiding local minima. Reward heuristics are used to make progress in highdimensional spaces. The resulting planner is applicable to general robots and manipulation tasks. We only assume the availability of a simulator for dynamics and gradients, e.g., Mujoco [44] . The search algorithm is stated in Alg. 1, and the components of the planner are introduced in the subsections below. Ablations for the planner's parameters are given in Appendix C. 
goal sampling: 
 Manipulation settings have highly constrained non-convex search spaces with many local minima. Inspired by rapidly exploring random trees [26] , n g different goal states are sampled during the search to escape local minima and enable global exploration. A bias b g ∈ [0, 1] is set to sample with a certain probability the actual goal for a given task s task . Otherwise, a random state is sampled uniformly in a box s random ∈ [s min , s max ] with upper and lower bounds s min and s max :s g = s task if p g ≤ b g s random otherwise, with p g ∼ U(0, 1), s random ∼ U(s min , s max ).Sampled non-unit-length orientation representations, i.e., quaternions, are normalized. Infeasible states, i.e., states with contact penetration, are discarded and resampled.Exploration in high-dimensional spaces is challenging due to the combinatorial explosion. Thus, each node in the tree is assigned a value to permit a heuristic selection of promising nodes during the search, similar to Monte Carlo tree search [45] . Instead of a discounted value, directly using a node's reward yields satisfactory results for us and removes the need for value back-propagation up the tree. When the goal state s g is changed, the total reward (value) of a node with state s,EQUATIONis computed from the three following reward components.Distance The actual closeness to the desired manipulation goal is formulated as a distance reward,EQUATIONwith weight matrix Q d ∈ R ns×ns . It encourages a small-scaled distance between a node's state and the goal. Note that for quaternions, the three-dimensional orientation difference is used [46] . EQUATIONwhere Q p ∈ R np×np is a weight matrix and d = [d 1 , . . . , d np ] T ∈ R np consists of all stacked sensor values. Close proximity between the robot and the object ensures frequent whole-body interactions but may prevent exploration because moving away from the object to change the robot's configuration is penalized. The weight matrix Q p is responsible for achieving a good trade-off.Reachability When being close to an object, a robot needs to be in a configuration where moving the object towards the goal is easily possible. Given an object's state difference from the goal ∆s o = s o -s g,o , a measure m for the difficulty of reaching the goal can be calculated with the local Mahalanobis metric defined in [14] asEQUATIONHere, B o is the control Jacobian of the object dynamics f o with respect to the action a, µ ∈ R is a regularization parameter to ensure the invertibility of M , and I ∈ R no×no is the identity matrix. A small m indicates that moving the object towards the goal is easy in the current robot configuration.The reachability reward based on m is defined asEQUATIONwith weight q m . In practice, m can be clipped to avoid overly large rewards (cf. Appendix A).At the beginning of each of the n i iterations, a node with index i n ∈ N + must be selected. A tradeoff is required between selecting the currently best node for greedy exploitation and selecting worse nodes for exploration. To this end, the tree's n n nodes are ranked from highest to lowest reward, and the index i n = [x ∼ P(n n , β)] is the rounded sample from a truncated Pareto distribution [47] with bounds [1, n n ] and exponent β ∈ R + . Larger values for the parameter β lead to a more greedy node selection. After the first node is selected, given the extension horizon n e , the remaining n e -1 node indices for the current iteration are the last extended nodes' indices. In practice, β and n e can be varied when the search is progressing well or poorly (cf. Appendix A).Exploration is based on a mix of globally exploring random actions to avoid local minima and locally optimal actions for precise fine-grained manipulation. Once a node is selected, a step size ∆t a and an action a = â•α is generated with element-wise multiplication of the direction â ∈ {R na | ∥â∥ = 1} and magnitude α ∈ {R na | 0 ≤ α ≤ α max }, where α max is the maximum magnitude. The action corresponds to a relative position command and is calculated using one of the methods listed below. The method used is sampled according to a user-specified discrete probability distribution p a . In practice, the step size ∆t a for each action can vary (cf. Appendix A).Random To ensure global exploration, a random action can be taken. This action samples a uniformly distributed direction â, and the magnitude α is sampled uniformly within the magnitude bounds.Continuation Obtaining a promising search direction is challenging, so instead of discarding a successful action after one step, it can prove beneficial to keep going in this direction. Accordingly, the continuation action takes the same action direction that leads to the currently selected node, i.e., âin = âjn , where i n and j n are the current and parent node indices, respectively. The magnitude α is sampled uniformly within the magnitude bounds.Proximity As explained for the proximity reward, closeness of robot and object is necessary for manipulation. Therefore, the values d of the virtual proximity sensors are also used to compute the proximity action. The action direction aims to minimize 1 2 d T d and points in the opposite direction of the gradient g = (∂d/∂a) T d, i.e., â = -g/∥g∥. The magnitude α is sampled uniformly within the magnitude bounds.Goal-direction Besides random actions for global exploration, goal-directed actions are required to make meaningful progress. Locally-optimized actions are computed by linearizing the dynamics around the reference values s * 0 = s 0 and a * 0 = a 0 -∆a 0 , with the fixed reference state s 0 :EQUATION)The linearized dynamics are used to solve the one-step optimal control problemEQUATIONwith a closed-form solution for ∆a 0 (cf. Appendix A). The resulting action direction is â = (a * 0 + ∆a 0 )/∥a * 0 + ∆a 0 ∥, where a * 0 can be chosen as, for example, the action used to reach the current node. The action magnitude is set to the maximum value, i.e., α = α max .Once the action is computed for a selected node, a dynamics rollout is performed. The reached state is appended to the search tree as a new node. Details on the rollout are given in Appendix A. 
bootstrapping reinforcement learning with demonstrations: 
 Our planner can find solutions for challenging manipulation tasks but provides demonstrations only for a single start state. Accordingly, we do not deploy the planner directly as an online policy but instead use the demonstrations to bootstrap reinforcement Learning (RL). This approach allows us to generalize beyond the planner's demonstrations and perform additional online exploration to exceed the planner's suboptimal performance stemming from partially random actions. By distilling a policy, we also no longer require the planner during policy training or execution. For the best inclusion of demonstration, we investigate several different methods, which can be grouped into the two main strategies outlined below. Further details are provided in Appendix B. Foreshadowing the results, we find that using a fixed ratio of demonstrations in the replay buffer provides the best performance of all investigated methods. This result aligns with the core idea of off-policy RL [48] and findings in [22] . It is a welcoming result due to the simplicity of implementing this strategy.During each training epoch, n r = n p + n π rollouts, consisting of n p offline planner demonstrations and n π online policy rollouts, are appended to the replay buffer. Afterward, training data is sampled randomly from the entire replay buffer. Similar to [22] , we investigate three different approaches for filling the replay buffer: (1) using a fixed ratio n p /n π in each epoch, (2) using a decreasing ratio as the policy's success rate increases, and (3) adding only planner demonstrations in the first epoch and only online policy rollouts afterward.Following the concepts outlined in [25] , we investigate three different approaches for using demonstrations for pre-training: (1) pre-training an imitation policy π IL from demonstrations and using it as a fixed policy alongside the RL-trained policy π RL , (2) pre-training π IL and initializing the value function Q from demonstrations, and (3) pre-training π IL as well as initializing Q and additionally adding a fixed ratio of demonstrations to the replay buffer as described above. 
experiments: 
 Given our motion planning and learning setup, we investigate three main questions: A) Can the planner generate solutions for dexterous and whole-body manipulation tasks? B) Can the solution demonstrations bootstrap manipulation learning? C) Can the learned policies be transferred to real systems? These questions are evaluated for the six tasks displayed in Fig. 1 and 2 . Note that additional tasks, baseline comparisons, and ablations, as well as details on systems, parameters, and experiment setups, are given in Appendix C. For each evaluation run, five random seeds are used.A) Manipulation planning The planner's ability to solve the manipulation tasks is measured as the search progress = 1 -r d (s t , s g )/r d (s 0 , s g ), i.e., the scaled relative closeness to the goal (cf. ( 6)). This progress is plotted for the six tasks in Fig. 2 . Despite the varying tasks and complexity of the settings, the planner consistently finds solutions for all scenarios, demonstrating that the rewards and actions are general enough to treat high-dimensional spaces and multi-contact dynamics. We compare the following approaches for demonstration inclusion in Fig. 3 : pure DDPG as a baseline, using a fixed ratio of n p /n r = 25% demonstrations for the replay buffer (cf. Sec. 5.1), and pre-training an imitation policy as a secondary policy (cf. Sec. 5.2). For comparability with the planner, the learning progress is plotted over the total actions taken. Using demonstrations always leads to an improvement over pure reinforcement learning, sometimes drastically, while pre-training an imitation policy performs inconsistently. Even though the demonstrations are limited to a single start state, they provide enough reward signals to learn successful policies faster and more robustly for all runs. At the same time, the demonstrations are not good enough to directly train and use an imitation policy, as adding this policy does not yield satisfactory results as it is slower and does not make any learning progress for some of the runs. We transfer policies from training in simulation directly to the hardware for the three tasks shown in Fig. 1 . For each setting, we pick a policy with good performance in simulation but perform no additional training in the real world. Table 1 shows the evaluation success rates on an initial-state grid, and Fig. 4 shows the progression of the stool-lift task as an example.The trained policies for the box-grasp and stool-lift tasks work well, given the direct policy deployment. It is worth noting that in the stool-lift task, contact occurs on the robot arms, not just the  
conclusions: 
 We present a versatile planner for dexterous and whole-body manipulation tasks. The planner generates demonstrations that are used to bootstrap reinforcement learning, leading to significant learning improvements. It is robust against increasing task complexity, making it a valuable tool for general and complex manipulation scenarios. Similarly, aiding reinforcement learning with synthetic demonstrations from the planner is simple to implement yet gives strong results, providing a promising architecture choice for general manipulation learning, also in combination with other methods.Limitations Our method has limitations. (1) Like any automated search, the planner requires hyperparameter tuning. However, the planner can provide results within seconds compared to learning algorithms that often require minutes or hours before success can be assessed. This fast process drastically reduces the iteration loop and allows manual tuning at interactive rates and quick parameter sweeps. (2) Both the planner and policies rely on state information, which requires a motion capture system in the real world to acquire such state information. There are several ways to extend this work to synthesize vision-based policies. One could transfer planner data to convert state-action trajectories to image-action trajectories. A second option would be to apply a teacher-student framework [49, 50, 51, 52, 38] by training a student policy that only has access to vision sensors to mimic our expert state-based policy. (3) Finally, we use a relatively simple pre-training setup, and the reported performance of imitation pre-training could change with more sophisticated IL methods. 
 
 Appendix: Implementation and parameters for the manipulation planner are given in this section.The three reward types of the manipulation planner are visualized in Fig. 5 .Figure 5 : Low and high reward states for the three rewards of the manipulation planner.The reachability measure m defined in ( 8) is clipped from below to a lower bound value m min ∈ R + in our implementation to prevent overly large rewards, i.e., m = max(m, m min ), and the reward calculation ( 9) is updated toEQUATIONA.The four action types of the manipulation planner are visualized in Fig. 6 . Varying action step size It can be helpful to have actions with different time step sizes to allow both short, precise motions and long, far-reaching motions. We randomly sample the action step size as an integer multiple of the base step size ∆t a , i.e.,EQUATIONwhere k max ∈ N is the upper bound for the step size scaling. After the planner's search is finished, we split all actions to the base step size to obtain one common action step size for learning.Closed-form solution for goal-directed action The closed-form solution for the optimal control problem (11) isEQUATIONDynamics rollout The actions of the planner are position commands tracked by a low-level PD controller u t = -K p (q r,t -a t ) -K d qr,t with controller gains K p and K d . A low-level control step size ∆t c ≤ ∆t a smaller than the action step size is used to generate a position referenceEQUATION)The reference is a linear interpolation between the last commanded position for the parent node, q r,jn + a jn , and the last actual position plus the newly computed relative action, q r,in + a in , which creates a continuous control input. The same control scheme is used for RL policy actions.The planner's parameters that are not task-specific are listed in Compute value targetv i = r i + γQ ′ ϕ ′ (s i+1 , π ′ θ ′ (s i+1 , s g,i+1), s g,i+1 )19:Compute value function gradientg Q for l Q = 1 n batch i (v i -Q θ (s i , a i , s g,i )) 2 20:Compute policy gradient g π for l π = 1n batch i Q θ (s i , π ϕ (s i , s g,i ), s g,i )21:Apply the update to π ϕ and Q θUpdate the target networks: 2 10: Compute policy gradient g π for l π = 1 n batch i ∥a i -π ϕ (s i , s g,i )∥ 2 11: Apply the update to π ϕ and Q θ A policy network π θIL of the same size as π θ can be trained with imitation learning from planner demonstrations. The policy is trained with input (s, s g ) and output (a) from the demonstrations. Similarly, the value function is trained with input (s, s g , a) and output (v), where the discounted value v for an input tuple can be calculated trivially from the rewards (1) for all states in a demonstration trajectory. Algorithm 5 states the pre-training procedure. Note that the policy and the value function can be pre-trained separately.ϕ ′ = τ ϕ + (1 -τ )ϕ ′ , θ ′ = τ θ + (1 -τ )for l Q = 1 n batch i (v i -Q θ (s i , a i , s g,i ))The pre-trained policy and value function are used as described in [25] . In summary, during the policy rollout and network training phase, the better one of the two policies, as evaluated by the value function, is used. Note that we use the same value function for pre-training and online training.The evaluation systems are described in this section. Afterward, several ablations and comparisons for the manipulation planner and reinforcement learning are provided.Table 4 provides the task-specific parameters for the evaluation systems. Note that all object velocity states have a scaling Q d,o,q = 0.1I and the robot state scaling is zero: Q d,r = 0I. A description of each system is provided below.The box push task is a one-dimensional pushing task where one actuated pusher box (robot) must push a second unactuated box (object) to a goal position on a line. Due to its simplicity, this task can give valuable insight into the general properties of planning and learning. Moreover, only pushing but no pulling is possible, which creates an interesting manipulation challenge. The two-dimensional box push task is the direct extension of the one-dimensional box push task. Just as the one-dimensional box push task, it is an illustrative example. The complexity of the task is increased by placing the pusher box between the object box and the goal. This setup requires the pusher to move around the obstacle before pushing it towards the goal.The planar hand is a simplified representation of in-hand manipulation in a twodimensional plane. A hand with two two-link fingers is used to rotate and lift a box. Even with the restriction to a plane, this task requires dexterous manipulation and gives insight into the planner's ability to solve in-hand manipulation tasks.The floating hand has the same kinematic as the planar hand and is also limited to a two-dimensional plane. However, instead of a fixed wrist, the floating hand can move in the plane. The task is to reorient a box and move it in space. This task combines dexterous manipulation as well as picking and placing an object, two commonly required skills.The Allegro hand is a robotic hand by Wonik Robotics with four fingers, sixteen actuated finger joints, and two actuated wrist joints. The task is to reorient a cube by turning it around the upright z-axis. In-hand manipulation is a commonly investigated robotic skill due to its high dimensionality and required dexterity.The box grasp task uses a stationary quadrupedal robot with a mounted arm to grasp and lift a box with a handle. It is a classic pick-and-place task, where finding a proper grasp is the main challenge. This task demonstrates our method's ability to solve pick-and-place scenarios on real systems.The ball turn task involves a quadrupedal robot on its back that is using its legs to reorient a ball. This setup resembles in-hand manipulation tasks where multiple multi-joint fingers (legs) are used to reorient an object. We use this setup to demonstrate our method's ability to perform in-hand manipulation on real systems.The stool lift task comprises two stationary quadrupedal robots with mounted arms that lift a bar stool into an upright configuration. The robots are deliberately prevented from grasping the stool by fixing their grippers in a closed state. In this manner, whole-body and bimanual manipulation of the stool is required, with contact occurring on multiple parts of the robot arms. We use this task as an exemplary real-world cleaning scenario, where robots can help put moved or fallen furniture back into desired configurations. By forcing whole-body contact, we also demonstrate our method's ability to perform whole-body manipulation on real systems.We present additional search results for two tasks and several studies on the planner's components and parameters.Box push and box push 2D results Extending the results presented in Fig. 2 , we show the search progress for the two additional tasks, box push and box push in 2D. The results are in line with the reported results for the other tasks. A comparison of three manipulation tasks with increasing difficulty based on the planar hand is given in Fig. 8 . The tasks are (1) in-hand reorientation of a box by 90 • , in-hand reorientation of a box by 180 • , and throwing the box onto a shelf. The 90 • turn can be achieved by simply tipping over the box. For 180 • , regrasping is required. Throwing the box onto the shelf requires highly dynamic movements to overcome the gap between the hand and the shelf. We compare our planner to the quasi-static planner in [14] . Relative closeness to the goal for the best node in the tree so far vs. the total number of nodes (simulation steps).We outperform the quasi-static planner across all three tasks. It is important to note that, for the throwing task, the search progress does not fully capture performance. While our planner consistently succeeds in throwing the box onto the shelf, the quasi-static planner fails to find this solution.These results underscore the necessity of dynamic planning for advanced manipulation tasks and highlight our planner's superior capability to solve such tasks, thereby outperforming a state-of-theart planner.The extension horizon specifies how many sequential actions are taken once a node has been selected to extend the tree. We assume that tasks of different complexity require different extension horizons. We evaluate different but fixed extension horizons for all tasks to measure the influence on the planner's search progress in Fig. 9 . We also evaluate the automatically varying horizon that we use in our implementation as described in Appendix A.2. We measure the average closeness to the goal over the full search duration, which is larger for quicker search progress and higher final closeness. While a longer extension horizon tends to improve performance overall, there is no strong correlation. the same time, the varying horizon performs well in all cases with reduced variance compared to the fixed horizons. Pareto distribution exponent The Pareto distribution parameter β modifies how greedily nodes with high rewards are extended. We evaluate different but fixed Pareto parameters for all tasks to investigate how the parameter influences the planner's search progress in Fig. 10 . We also evaluate automatically varying parameter that we use in our implementation as described in Appendix A.2. For intuition, a parameter β = 0.05 means that the currently best node will be selected with a probability of roughly P = 10%, whereas β = 2.0 means a probability of P = 75%. There is a consistent trend that the planner's performance decreases as the node selection becomes more greedy (the parameter increases). This effect could be caused by the node selection becoming so greedy that the search gets stuck in local minima by repeatedly selecting the same few nodes. Using an automatically varying range for the parameter does not lead to an improvement over always choosing a low parameter. Apparently, the Pareto distribution with a low β already has a good tradeoff between exploitation and exploration.Sub-goal ratio To assess the usefulness of having sub-goals during the planner's search, as in rapidly exploring random trees, we evaluate different ratios of sub-goals to node extensions in Fig. 11 . A goals/extensions ratio of 1/900 means we choose one goal and do 900 extensions for this single goal, whereas a ratio of 900/1 means we choose 900 goals and do one extension for each of these goals. Note that in this evaluation, we always do a total of 900 extensions. There is no consistent trend indicating that more or less sub-goals improve search performance. Multiple explanations exist for these results: Our tasks could have very few local minima. However, since the Pareto parameter does influence search performance (cf. Fig. 10 ), there may be many local minima, but our node selection strategy is already good enough to avoid these. Yet another possibility is that the difference in sub-goals is too small to provide a benefit over searching with just a single goal. We investigate the effectiveness of different combinations of action types with just random actions as a baseline in Fig. 12 . The fact that purely random actions already lead to good results in many tasks indicates that the other components of the planner, such as the heuristic node selection and specific rewards, lead to meaningful search progress. Still, proximity and goal-directed actions alongside random actions in the search further improve the performance for some tasks.Proximity and reachability reward scaling The effect of different proximity and reachability reward scalings is shown in Fig. 13 . For most tasks, the search fails once the proximity reward becomes too high, potentially because the necessary distance between the robot and the object for , proximity+goal-directed (pg), and random (black).reconfiguration is punished too severely. The reachability reward scaling shows a similar, although weaker, trend. Especially for the more complex tasks involving the Allegro hand or robot arms, it is difficult to clearly state how to scale the rewards, and a parameter sweep is necessary to find the optimal value. It can be assumed that the same effect would occur for dense-reward reinforcement learning, highlighting one of the benefits of the planner, as parameter sweeps can be executed much faster with the planner than with learning. We present additional learning results for two tasks and several comparisons for different learning methods and demonstration usage.Box push and box push 2D results Extending the results presented in Fig. 3 , we show the training progress for the two additional tasks, box push and box push in 2D. The results are in line with the reported results for the other tasks. Figure 14 : The performance of reinforcement learning for manipulation on two additional tasks. Comparison of our approach of adding demonstrations to the replay buffer (blue), RL with an additional imitation policy pre-training (red), and pure RL (black).While the planner is able to solve the manipulation tasks presented in this paper, distilling a policy from planner demonstrations leads to more optimal and faster solutions (after training). The trajectory lengths (in steps) for planning and learning are given in Tab. 5. We list the average trajectory lengths for five seeds.The results show that both the planner and the trained policies are able to find solutions. However, the policies always require significantly fewer steps to solve the tasks, indicating a more optimized performance.The wall-clock times for planning and learning, as well as dimensions for the manipulation tasks, are given in Tab. 6. We list the average time for five seeds. Single sol. refers to finding a single trajectory leading to one goal. Multiple sol. refers to finding multiple solutions leading to different goal locations in order to generate a set of demonstrations for policy learning. Policy training refers to training the policy where demonstrations are fed to the replay buffer. The planner is no longer used at this stage. Observation dim. refers to the combined dimensions of state and goal space. Action dim. refers to the dimension of the action space.The results show that both planning and policy learning scales to high-dimensional tasks with only a moderate increase in computation time.We provide a comparison to the state-of-the-art manipulation learning algorithm MoPA-RL [37] for the same three planar hand tasks described in Appendix C.2.2 and for a complex assembly task. Planar hand We use sparse rewards for the baseline comparison on the planar-hand tasks and show the results in Fig. 15 . We outperform MoPA-RL for all three tasks. While the MoPA-RL is also able to quickly and consistently find solutions to the 90 • -reorientation task, it has more difficulty with the 180 • reorientation. Finally, MoPA-RL is not able to find any solution to the throwing task, whereas our method achieves a success rate of more than 50%. As before, the results demonstrate the need for dynamic planning and our planner's ability to solve highly dynamic tasks.Sawyer assembly With the Sawyer assembly task described in [37] , we demonstrate our ability to solve complex assembly tasks that require precise manipulation in cluttered environments. The task is challenging because colliding with the movable table prevents assembly. Accordingly, collisions must be avoided. We evaluate five seeds for our method, and we reuse the results shown in [37] for MoPA-RL. The performance of our planner for solving the Sawyer assembly task presented in [37] . Relative closeness to the goal for the best node in the tree so far vs. the total number of nodes (simulation steps). Right: Performance comparison of our full algorithm with MoPA-RL [37] for the Sawyer assembly task. Results for MoPA-RL are taken from [37] .In the majority of cases, our planner is able to find solutions for this assembly task. Policy learning is successful in these cases with good planner demonstrations. While we achieve a similar average performance for policy learning as MoPA-RL, we have a higher variance due to the reliance on the planner's demonstrations. While our planner is designed for contact-rich manipulation, it can also solve this assembly task where contacts must be avoided. Our method implicitly takes collision avoidance into account and matches the performance of MoPA-RL.Demonstration ratio Adding demonstrations to the RL replay buffer improves performance compared to pure online learning, but pure offline learning with the demonstration data does not yield good results. Accordingly, there must be a tradeoff between online exploration and offline data. We investigate this tradeoff by varying the ratio of demonstrations in the replay buffer in Fig. 17 . We measure the average reward over the entire training duration, which is larger for quicker learning progress and a higher final success rate. It appears that a small amount of demonstrations gives enough reward signals to bootstrap RL and enable exploration from a sub-optimal solution. Already having a low ratio of roughly 10% yields satisfactory results, and we use 25% as it gives the most robust performance for all tasks. Demonstration inclusion comparison Three different methods for adding demonstrations to the replay buffer are investigated. (1) Use a fixed ratio of demonstrations to online exploration. (2) Use a variable ratio, where the ratio decreases as the learning success rate increases. (3) Add a fixed number of demonstrations from the planner's tree only initially. As can be seen in Fig. 18 , the specific method of introducing demonstrations into the replay buffer does not make a large difference. However, just adding demonstrations in the beginning is sometimes not enough to bootstrap learning successfully.Pre-training comparison Directly using the pre-trained policy only works for the most simple box push task with a success rate of 0.10 ± 0.13. For all other tasks, the success rate was zero. We evaluated ten runs for each task. Apparently, the data generated by the planner is not good enough to directly perform imitation learning with it, and online exploration with RL is required to find successful policies. (3) Pre-training the policy and value function as well as adding a fixed ratio of demonstrations to the replay buffer. While adding a pre-trained policy to RL improves performance over pure DDPG, it does not significantly and consistently perform much better. Only once demonstrations are added to the replay buffer does the performance with a pre-trained policy improve significantly. Since the performance of adding demonstrations alongside pre-training is not better than just adding demonstrations, we attribute this improvement to using demonstrations and not to the pre-trained policy.HER comparison For reference, we provide a comparison to hindsight experience replay (HER) in RL since this method can improve learning performance in sparse-reward settings. The results are shown in Fig. 20 . While using HER improves the learning performance for some tasks compared to pure DDPG, it does not lead to any improvement for others and almost always performs worse than adding demonstrations to the replay buffer.Setup We use the quadrupedal Spot robot by Boston Dynamics with an arm arm mounted for certain tasks. Control and communication with the robot are achieved via the bosdyn software development kit. Accurate state estimation is achieved by using the OptiTrack motion capture system and placing markers on the objects. Communication between the Optitrack System and the robots was handled via ROS2, while gRPC was used to command policy actions with the Spot robots. The performance of reinforcement learning for manipulation with hindsight experience replay on eight tasks. Comparison of our approach of adding demonstrations to the replay buffer (blue), RL with 50% HER relabeling (light green), RL with 80% HER relabeling (dark green), and pure RL (black).The box grasp task evaluates if the robot is able to grasp and lift the box. A trial is considered successful if the box is lifted off the ground. We evaluate 27 different starting locations by building a grid of start positions and orientations for the box:x ∈ {-0.1, 0.0, 0.1} × y ∈ {-0.25, 0.00, 0.25} × θ z ∈ {-45 Fig. 21 shows the progression of the box-grasp task. The challenge in this task is to learn that in order to lift the box, the handle must be grasped. Our trained policy is able to consistently grasp the handle and performs well. Ball turn The ball-turn task is used as an in-hand-manipulation-equivalent example. A trial is considered successful if the box is rotated such that the angle deviation from the upright z-axis is below 40 • . We evaluate 20 different starting locations by testing a sequence of start rotations around the y-axis for the ball with increments of 15 • :EQUATIONFig. 22 shows the progression of the ball-turn task. The robot can successfully perform some of the desired orientations. We attribute failures to the sim2real gap, with a rigid ball in simulation training and a soft yoga ball in the real-system setting. Fig. 4 shows the progression of the stool-lift task. Despite the challenging whole-body and bimanual setup, the robots frequently manage to lift the stool in the upright configuration. Failures can be attributed to a lack of measuring the robots' body positions. When contact forces occur, the robots' bodies are displaced (which we do not measure), leading to position offsets for the robot arms. In such cases, the policy has false state measurements and produces unsuccessful actions.8th Conference on Robot Learning (CoRL 2024), Munich, Germany.If the robot state contains a quaternion, the joint velocity is lower dimensional than the joint configuration. 


Decision: Strengths: - This work integrates motion planners with RL to bootstrap learning and create an efficient learning approach that can be used in diverse applications like in-hand manipulation and lifting with robots. - Sim2real transfer of learned policies Weaknesses: - Lack of baselines: Authors must provide comparisons against baselines from literature including some mentioned by reviewers. How does the performance differ from a pure motion planning approach? Reviewers also raise concerns about the overall novelty of the proposed approach which could be addressed through writing and baseline comparisons too. - The heuristics used to get good demonstrations from the planner make the generalizability of the approach concerning. Authors should provide more motivation for why this is scalable to more tasks especially tasks that might be challenging for motion planning? - Dependence on state space in the real-world is constraining. How does performance deteriorate when using visual policies? 

Post-rebuttal discussion: The key points addressed in the rebuttal are: - Baseline comparisons and novelty: Authors have added comparisons with other approaches from literature and added a new dynamic throwing task in simulation. They have also added discussion about their approach versus others from literature aiming to better highlight their novelty. However the additional simulation comparison experiments are too simplistic and the paper could benefit from stronger baseline comparisons especially in the real-world, which we encourage the authors to include in the camera-ready version.

Reviews: review: title: title: Good paper to read, but the novelty of the paper needs to be justified
paper_summary: paper_summary: The motivation of this paper is that collecting human demonstration data is costly and not scalable. One would like to use motion planners to generate some demonstration data. However, the motion planner often runs under the assumption of quasi-static dynamics, and the resulting demonstration data might lack action labels. To address this challenge, the paper proposed to use the gradient descent method that computes the gradient through the planner. The resulting generated data is used to train control policies. The goal is not to develop a method that improves the solution-finding capability of the planner but to use that dataset to bootstrap the learning policy.  To find the solution given the dynamics or simulation of the robot, the paper proposes to use tree search methods to find a good planner solution for the whole-body manipulation. The process goes through (1) goal sampling, (2) reward update, (3) node selection, (4) action selection, and (5) transit to the next state given action and the current state.   The second half of the paper talks about how to use such data to train RL policies. The first approach is to add data to the replay buffer, and the second approach is to pre-train the Q function on such an offline dataset.  Finally, the proposed approach is tested in real-world robot tasks.
main_review: main_review: Strength
1. The paper is easy to read, and Figure 1 helps the reader to understand the proposed method.
2. The proposed method makes sense in the sense that one can pre-train policies with a handful of trajectory data from the planner. And the planner here is another form of demonstration.

Weakness
1. One concern is that the RL policies' performance is bounded by the performance of the planner. In the paper, the authors do not compare the quality of the resulting trajectories to other methods that solve the optimization problem of the planner. It would be nice to comment on the quality of the trajectory and its effect on the RL policies.
2.  Another issue is that if we can use a reliable planner and solver to get a good trajectory, why not just deploy such a method in the real world? And the benefit of neural networks is the ability to generalize the unseen setting. I think the experiment section does not have experiments in which you show the planner solver fails to find a solution while the RL policies do not. It would be nice to hear more thoughts on this from the authors.
3. Using tree-search to find solver solutions is not new -- it could be traced back to MCTS or Alpha-Go-like system. Recently, there have been papers that propagate the gradient to the entire solver. As a result, the seem to lack novelty.
questions: questions: 1. Could authors give an example of how fast the proposed method can run (wall clock time) under what dimension of the states and actions?
limitations: limitations: 3
review_summary: review_summary: For the above weakness, the paper is on the borderline.
title: title: interesting paper
paper_summary: paper_summary: the paper propose a planner for boostrapping RL for manipulation tasks
main_review: main_review: Overall, this is an interesting paper with good demonstrations, with interesting sim2real experiments.

 In general, the proposed method relies on search which seems to require quite a bit of front loading effort but it can be applied in a variety of tasks like performing in-hand manipulation and lifting a stool using two spot robots. The front-loaded efforts include quite a bit of hand-crafted engineering for constructing the rewards (distance, proximity, reachability), node selection criteria (i.e. possible actions). Then, further implementation is needed to boostrap these demonstrations for RL. These learned policies can even be transferred from sim2real, which is nice, however, these are mainly for rigid objects for which there might be less sim2real gap. Still, performing tasks like lifting a stool with two spots (which might be quite challenging to teleop for imitation learning) is still great. 

The major limitation, as the authors point out, is the need for mocap for observing the full state of the objects, which is not ideal for real-world scenarios. There also seems to be missing comparison to existing state-of-art algorithms.
questions: questions: - Can you compare to other previous state of art algorithms? Or some way to show improvement of your method beyond existing state of art algorithms? This seems possible in sim, potentially.
limitations: limitations: 3
review_summary: review_summary: strong real-world experiments and the technical details seem sound
title: title: Official Review
paper_summary: paper_summary: This paper proposes a flexible and general manipulation planner that can generate dynamically feasible trajectories for dexterous and whole body manipulation. The planner is used to generate demonstrations for guiding a reinforcement learning agent in simulation. The learned manipulation policy is then deployed zero-shot on a real robot system. The overall pipeline is evaluated on manipulation tasks showing both efficient learning and reasonable sim-to-real transfer.
main_review: main_review: **Strengths**
1. The high-level motivation of the paper is well taken. I agree that model-based planners can play a big role in guiding reinforcement learning agents. While we have seen good results from using human-generated demonstrations to guide RL, model-based planners can certainly play a bigger role.
2. The paper is reasonably clearly written.
3. The investigation of different ways to combine planner generated demos with RL generated transitions is informative.
4. Unsurprisingly, the proposed pipeline leads to significantly better learning on a number of interesting manipulation tasks. I particularly like the transfer of policies to real robots.

**Weaknesses/Comments**
1. My biggest concern with this paper is that it does not compare the proposed manipulation planner with any baseline whatsoever. While I understand that many manipulation planners make simplifying assumptions about the system dynamics, it is still worthwhile to compare with them. For example, I believe some of the tasks, such as, box-grasp and stool-lift, could be solved even with a quasi-static assumption. An experimental comparison could better buttress their point about the need to plan with full dynamics.
2. This paper misses highly relevant prior work that do or could potentially generate dynamically feasible trajectories (see [1] and [2]). The proposed planner seems similar to these methods in that it combines tree search with a set of actions generated heuristically or using trajectory optimization. Hence, I am not sure about its novelty and significance.
3. The planner has no theoretical properties of completeness or suboptimality.

[1] Natarajan, Ramkumar, et al. "Torque-limited manipulation planning through contact by interleaving graph search and trajectory optimization." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

[2] Cheng, Xianyi, et al. "Enhancing dexterity in robotic manipulation via hierarchical contact exploration." IEEE Robotics and Automation Letters 9.1 (2023): 390-397.
questions: questions: 1. Please experimentally compare the proposed planner with baselines, even if they make the quasi-static or quasi-dynamic assumption. For example, the authors have cited [14] and [15].
1. Please clarify how the planner compares with and differs from [1] and [2] (see review). Ideally I would like to see an experimental comparison but a discussion may suffice if the authors can clearly articulate how their method is superior.
limitations: limitations: 3
review_summary: review_summary: The paper makes two contributions: development of a model-based manipulation planner and using the planner to guide an RL agent. I am unable to recommend acceptance because I am not convinced of the novelty and significance of the planner due to a lack of baselines.
title: title: The paper proposes to use motion-planning for bootstrapping several dextrous and whole-body manipulation tasks
paper_summary: paper_summary: This paper proposes a two-stage framework for learning manipulation policies, wherein the first stage, motion-planning is used to generate expert demonstrations removing the need for collecting human-generated expert trajectories. In stage 2, these demonstrations are used to augment the replay buffer of an off-policy deep reinforcement learning for bootstrapping the learning process for various dexterous and whole-body manipulation tasks.
main_review: main_review: The paper proposes an interesting method for bootstrapping learning for complex manipulation tasks using motion planners as experts providing demonstrations. The paper is clearly written and well-organized. The paper also provides several simulation and real-world experiments, showcasing the efficacy and practicality of the proposed approach. While the results are mostly impressive, it is unclear if using motion planning techniques is scalable as well as generalizable to a lot of tasks, due to its non-real time computational delays as well as the dependency on access to the state space. Moreover, the final policy still relies on state information which is not readily accessible in the real-world, especially when deployed in-the-wild. Moreover, the requirement to predesign rewards for each of the state spaces makes it difficult to scale to a lot of manipulation problems with stochastic environments or where motion-planning typically fails such as where complex assembly or precise manipulation is required.
questions: questions: I request the authors to address the concerns below:
1. As acknowledged by the authors, the dependency of both the planner as well as the policy on the state space creates a bottleneck for this approach to scale to diverse tasks in the real-world. Another related work [1] further distills the state-based policy into an asymmetric policy which still relies on the state space for the critic, but distills the actor into a vision-based policy, which is more readily deployable in the real-world outside of constrained environments. Can a similar technique be used here for learning visual policies? Moreover, [1] and [2] seem to be highly relevant to this work and must be included in the related works.

2. Motion-planning is classically beneficial for collision-free planning. Have the authors tried any tasks where motion-planning provides any additional benefits such as finding collision-free trajectories in cluttered environments? 

3. How can this method be extended to complex manipulation tasks which are non-trivial for motion-planning to solve, such as complex assembly tasks or dynamic object manipulation?

4. While there are several benefits to using classical motion planners, what are the challenges and bottlenecks of using them in terms of both computational complexity as well as real-time efficacy? It would be helpful to know how much time and resources are allocated to collecting expert demonstrations for each of these tasks.

5. It is unclear how the expert trajectories are collected for the collaborative task of stool-lifting with two arms. An elaborate explanation of that would be helpful.

[1] Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation. 

[2] Motion Planner Augmented Reinforcement Learning for Obstructed Environments.
limitations: limitations: 3
review_summary: review_summary: The paper is clearly written and easy to follow. While the problem formulation as well as the experiments are convincing, there are several open questions that, if addressed, will help the community to further take this approach to a generalizable and scalable solution.