DiffusionSeeder: Seeding Motion Optimization with Diffusion for Rapid Motion Planning

Authors: huang  huang
balakumar  sundaralingam
nvidia arsalan mousavian
nvidia  adithyavairavan
murali  nvidia
ken  goldberg
dieter  fox nvidia
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 02:43:58

Abstract: Running optimization across many parallel seeds leveraging GPU compute [2] have relaxed the need for a good initialization, but this can fail if the problem is highly non-convex as all seeds could get stuck in local minima. One such setting is collision-free motion optimization for robot manipulation, where optimization converges quickly on easy problems but struggle in obstacle dense environments (e.g., a cluttered cabinet or table). In these situations, graph based planning algorithms are called to obtain seeds, resulting significant slowdowns. We propose DiffusionSeeder, a diffusion based approach that generates trajectories to seed motion optimization for rapid robot motion planning. DiffusionSeeder takes the initial depth image observation of the scene and generates high quality, multi-modal trajectories that are then fine-tuned with few iterations of motion optimization. We integrated DiffusionSeeder with cuRobo, a GPU-accelerated motion optimization method, to generate the seed trajectories which results in 12x speed up on average, and 36x speed up for more complicated problems, while achieving 10% higher success rate in partially observed simulation environments. Our results prove the effectiveness of using diverse solutions from learned diffusion model. Physical experiments on a Franka robot demonstrate the sim2real transfer of DiffusionSeeder to the real robot, with an average success rate of 86% and planning time of 26ms, increasing on cuRobo by 51% higher success rate and 2.5x speed up. The code and the model weights will be available after publication.

Main paper: 
 introduction: 
 Robot motion generation aims to find a collision-free path in robot configuration space to reach a desired goal pose from a starting configuration. It is challenging due to the high dimensional robot configuration space, where each configuration corresponds to a combination of robot joint space and an environment representation. This problem has been studied using sampling-based methods [3, 4] , which sample configurations in the configuration space and checks for collisions in-between until a complete path is found. Others have leveraged optimization-based methods [5, 6, 7] , which optimize an initial path under specific objectives and constraints, to solve this problem. For both approaches, planning time increases significantly with the complexity of the configuration space [8] , resulting in a large variance in planning time for different problems. Furthermore, sampling-based methods often require post-processing to smooth the generated paths, while optimization-based methods, which can generate smooth paths directly, are highly sensitive to the initial seed. Several optimizationbased algorithms [2, 9, 5] address this problem by starting from a set of seed trajectories for optimization. The quality of seed trajectories are crucial, as an absence of good seeds can lead to failure in densely populated environments. Therefore, modern implementations of planners are forced to resort to graph-based planning algorithms to guarantee completeness at the cost of speed.Recently, learning-based methods have demonstrated great promise at motion generation without access to ground-truth state, achieving faster planning than sampling-based methods [9, 10, 11] . Some work [8, 12, 11] predict the next joint state iteratively instead of the entire trajectory, increasing the planning time. Learning-based methods often lack theoretical guarantees, fail to generalize to out of distribution scenes, and tend to generate sub-optimal plans as we later show in our results.We aim to develop a rapid and practical off-the-shelf motion planning pipeline that works with partial observations. We propose DiffusionSeeder, which generates diverse, high-quality paths to seed the optimization based motion planner, as shown in Figure 1 , significantly reducing both the seed trajectory generation time and the number of optimization steps required. Denoising Diffusion Probabilistic Models (DDPM) [13] , a class of generative networks that model the output generation as a denoising process, have shown success in capturing complex multi-modal distributions in image generation and, recently, in robot manipulation [14] . Robot motion planning is well known to be a multi-modal problem, making DDPM a great fit for seed trajectories generation. Diffu-sionSeeder uses a conditional DDPM, consisting of an observation encoder and a noise prediction network for seed trajectory generation. We represent the partially observed environment with depth observations, which are faster to process than point clouds [11, 8, 12] . We train DiffusionSeeder on 3M simulated scenes from MπNet [11] . We use cuRobo, a fast motion optimizer utilizing GPU hyper-parallelization, to optimize DiffusionSeeder generated seeds for final trajectories. This paper makes the following contributions:• DiffusionSeeder, a conditional DDPM that generates high-quality seed trajectories within 6 milliseconds from a raw depth image, camera pose, start joint configuration and end pose.• A robot motion planning dataset consisting of 15M smooth and fast trajectories generated by cuRobo on 3M MπNet [11] scenes in simulation.• Integration of DiffusionSeeder with cuRobo, achieving 12× average speed up over standard cuRobo (and 36x speed up at 98th quantile), while achieving a 10% higher success rate on partially occluded views of a scene. 
related work: 
 Motion planning for robotics has been broadly explored. Search-based planning methods such as A* [15, 16] are complete, fast, and optimal for problems with discrete state spaces. Sampling-based methods such as RRT [3] , and PRM [4] are asymptotically optimal but can generate sub-optimal and erratic trajectories in practice with limited compute budget. Optimization-based methods [5, 6, 7] are susceptible to local minima in complex problem spaces. GPU-parallelized computations have been leveraged to address this local minima issue [2] . cuRobo [2] is a GPU-accelerated motion optimizer that solves on many seeds with linear interpolations from start to goal configurations in parallel to avoid local minima. In the case of failure, it falls back to a graph based motion planner to generate new seeds which significantly slows down the planning time, taking up to 0.5 seconds. Our work aims to mitigate this issue by using a diffusion model to generate high quality multi-modal seed trajectories, resulting in fast convergence and removing the need for graph based planner.Many works have utilized a learned neural network for generating environment-dependent sampling distributions [17, 9, 10, 18, 19] , accelerating sampling-based planning by biasing samples toward collision-free joint configurations. But the planning time still scales with the environment complexity. Other approaches learn a collision model [20, 21] for fast collision checks, but still rely on exhaustive sampling with Model Predictive Path Integral (MPPI) [22, 23] and a local controller.Other works [11, 8, 12] have formulated the motion planner as a policy which is queried sequentially to generate a trajectory. Motion Planning Networks (MPNet) [8, 12] and MπNet [11] learn policies to directly predict the next joint configuration from a point cloud. However, motion planning policies require multiple rollout steps to generating one trajectory, resulting in longer planning time.There is no further optimization on the generated trajectory, which can result in suboptimal trajectories. Some approaches, such as Ichnowski et al. [24] , warm-start optimizers with network-generated seeds, but lack generalization to new scenes and output only one trajectory. Yoon et al. [25] learn a reinforcement learning policy to generate an initial trajectory, but multi-step rollouts are slow and limited to one result, while DiffusionSeeder can generate multiple diverse trajectories quickly.Diffusion models have been leveraged for trajectory optimization. Janner et al. [26] trains a diffusion model per environment, making it impractical for diverse scenes. Carvalho et al. [27] and Saha et al. [28] train an observation/environment agnostic diffusion model and leverage the problem costs to guide the diffusion sampling to generate trajectories priors satisfying the scene-specific constraints, which may struggle to generate collision-free trajectories in partially observed cluttered environments. DiffusionSeeder is conditioned on an environment encoding and can generate environmentspecific trajectories for various environments, which are more efficient for motion planning in complicated environments. Huang et al. [29] represents the environment as point clouds and utilizes goal-oriented planning for robot motion planning, formulated as a motion in-painting problem, which requires multiple policy rollout steps for generating one trajectory. DiffusionSeeder takes in depth observations as the input to a Vision Transformer (ViT), in contrast to slower-to-process point clouds. Further, DiffusionSeeder generates multiple full trajectories for each inference pass of the diffusion model, avoiding multi-step policy rollouts. 
problem statement: 
 We study the robotics motion planning problem. The goal is to generate a fast, smooth and collisionfree trajectory τ to bring a robot from a starting configuration to a goal pose, given a depth image of dimension H × W , the calibrated intrinsic and extrinsic matrices of a third-person view camera, the starting robot joint state, q 0 , and the goal robot end effector pose (a 7d vector of a translation and quaternion in Cartesian space). We aim to develop a seeder that can generate diverse and high quality seed trajectories to accelerate a motion optimizer to rapidly generate a collision-free trajectory τ with position and rotation errors below given thresholds (δ t and δ r , respectively), while minimizing the jerk and motion time of the generated trajectory. We define τ to be {(t i , q i )|i ∈ [0, T ]}, where t i is the time step, q i is the robot joint state at time step t i , and T is the length of the trajectory. 
dataset generation: 
 We generate our dataset using cuRobo for a 7-DoF Franka robot on 3M MπNet training scenes [11] in three categories: cubby, tabletop and dresser (Appendix Fig 5) . A motion planning problem in the MπNet dataset consists of a scene mesh geometry, a start and end pose. To increase data diversity, we sample additional pairs of start and end poses based on the original start and end poses. Problems for which cuRobo is unable to find a feasible solution are discarded. We generated 15M problems with feasible and smooth solutions of size 32×7 on 3M scenes, where 32 is the trajectory length T and 7 is the robot DoF. More details are in Appendix 7.1.To provide partial observations to the planner, we render depth images for each scene using a fixed virtual camera located within the scene. To generalize to different view points, we sample 12 different camera poses for each scene. We render depth images from each camera pose and filter out camera poses where the problem end pose is not visible in the rendered depth images, resulting 12M rendered depth images of size 256 × 256 on 3M scenes. We save the corresponding camera intrinsic and extrinsic in robot base frame for each depth image. More details are in Appendix 7.2.DiffusionSeeder uses a conditional DDPM, consisting of an environment observation encoder and a conditional noise prediction network, to generate diverse seed trajectories, which can be optimized by an optimizer cuRobo for the final trajectory, as shown in Figure 2 .The observation encoder processes depth images, camera poses, the start joint configuration, and the goal pose to create a single embedding vector as the condition of the denoising diffusion process. Specifically, we use a ViT with 12 layers and 12 heads as the network backbone. Depth images, projected to 3D using camera intrinsics, are divided into 64×64 patches where each patch is encoded first and concatenated with the linear projection of the SE(3) homogeneous transformation of each camera pose. We add a positional embedding to each patch embedding, resulting a single 512dimensional visual embedding using the class token of ViT. The visual embedding is concatenated with the 64-dimensional encoding vector of the start configuration and the goal pose, resulting in a 576-dimensional environment embedding vector for the conditional noise prediction model.We adopt the CNN based architecture from [14] as the conditional noise prediction model, which is a 3-level UNet [30] architecture consisting of conditional residual blocks with channel dimensions [256, 512, 1024]. The model encodes the time step into a latent vector of 256 dimensions through a multi-layer perceptron (MLP), which is concatenated with the environment embedding from the observation encoder, resulting an 831-dimensional conditional vector.We jointly train the observation encoder and noise prediction model of the DDPM. During training, for each problem, we sample one depth image from the pre-rendered depth images. At each training iteration, we sample a ground truth trajectory τ from the dataset. We randomly select a denoising step k ∈ [0, K] and a random noise ϵ, which is added to the ground truth trajectories, resulting a noisy trajectory τ = τ + ϵ. The objective of the noise prediction network Θ is to predict the noise added to the original trajectory, with the training loss defined asEQUATIONwhere Φ is the observation encoder and O is the observation.In robot motion planning problems, the error in each joint can have different impact on the end effector error, due to the non-linearity of the robot forward kinematics (FK) model. To mitigate this issue, we pass both ϵ and the predicted noise ε to the robot FK model to reflect this non-linearity. We use FK to obtain the position of many points sampled on the robot's geometry across all links. We calculate the distance between the predicted positions of these points and the labels (using FK on joint state) as the loss. This loss gives a more direct representation of the error in the Cartesian Space for the whole robot. In addition, as many of the motion planning problems can be solved through a linear interpolation solution, we upweight the loss for non-linear solution to encourage the model to pay attention to the non-linear solutions, which are often harder to solve. As cuRobo will only call a graph based planner when the linear interpolation solution has failed, we use this to approximate the non-linearity of the trajectories. Together, we define our training loss as We use Denoising Diffusion Implicit Models (DDIM) [31] for faster inference. Combined with CUDA optimization, we achieve a 6ms inference time on an Nvidia GeForce RTX 4090, which includes the inference of the vision encoder and 5 steps of denoising.EQUATIONWe incorporate the trained DiffusionSeeder with an existing trajectory optimizer available in cuRobo. Joint space trajectories τ from DiffusionSeeder, the robot start configuration q 0 , target end pose X d are passed to the optimizer to optimize for N iters iterations. Details are in Appendix 7.6.In partially observed scenes, for optimizer collision checking, we construct a Euclidean Signed Distance Field (ESDF) using nvblox [1] from input depth images. Nvblox is a GPU-accelerated signed distance field construction library designed for robotic path planning, which uses GPU parallel computation to efficiently perform volumetric mapping of the world, while also enabling sharing of the generated map in a zero-copy mode with cuRobo for trajectory optimization on the GPU.  
experimental evaluation: 
 We conduct experiments both in simulation and on a physical Franka Panda robot to evaluate Dif-fusionSeeder. In the following sections, we denote DiffusionSeeder as DiffusionSeeder combined with cuRobo, unless otherwise specified. We use the same set of parameters for both simulation and real-world experiments. We run DiffusionSeeder with N denoising denoising steps to generate trajectories of length T = 32 in the joint space. For each problem, we pass N trajs randomly sampled noise with the same condition from the observation encoder to the diffusion model to generate N trajs initial trajectories for optimization. N denoising = 5, N trajs = 12 achieves good trade off between generation quality, diversity, and inference time. We consider the following quantitative metrics: success rate, plan time, jerk, translation error δ t , quaternion error δ r , and motion time (See Appendix 7.7). All metrics are computed over successful trajectories. We evaluate DiffusionSeeder on the MπNet simulation test set of 1791 problems. Each problem has a scene from the same scene types as the training data but with different configurations (Figure 5 ). We compare DiffusionSeeder to BiTStar, MπNet and cuRobo(v0.6.2). We use δ t = 0.005m and δ r = 2.86 • as the success threshold for both cuRobo and DiffusionSeeder as in [2] .We represent scenes with a single depth image. Both cuRobo and DiffusionSeeder use nvblox with the given depth image for collision checking, which takes 30ms to reconstruct the ESDF. The total time of DiffusionSeeder and cuRobo is the sum of the planning and nvblox reconstruction time. In practice, this can be expedited by running nvblox construction in parallel with the diffusion model inference. For BiTStar, mesh generation from the depth image takes 0.17s. For MπNet, we project depth images to point clouds using the camera intrinsics. We evaluate the generated trajectories from all method using the ground truth mesh for collision checking (see Appendix 7.8).We run cuRobo for a maximum of N atp attempts, denoted as cuRobo-N atp . For each attempt, we check if the trajectory meets the success criterion, repeating until a successful trajectory is generated or the attempt limit is reached. We use N atp = 1, 10, 100 as more attempts does not further increase the performance. For DiffusionSeeder, we run N iters optimization iterations for the diffusion generated trajectories, denoted as DiffusionSeeder-N iters . We use N iters = 25, 50, 100, 200, 475, as more optimization iterations doesn't increase the trajectory quality much but increases the planning time.We evaluate all methods on a Nvidia GeForce RTX 4090.The average values of each metric are summarized in Table 1 . Experiments in fully observed environments using ground-truth meshes, ablations on N trajs and number of depth images, Diffusion-Seeder only without cuRobo and more discussions are included in Appendix 7.8.BiTStar [32] is a sampling-based planner that utilizes the benefits of the random geometric graph. It can generate high quality and optimal solutions quickly compared to other sampling-based methods, especially in high dimensional problems, and has the properties of probabilistic completeness and asymptotic optimality. We incorporate BiTStar from OMPL [33] into MoveIt2 [34] with a timeout of 5s and a maximum of 10 attempts per timeout. Since MoveIt2 constructs goal constraints using the Euler angle error on each axis, we set the angle error tolerance on goal pose to be 0.01 rad for each axis and the translation error tolerance to be 0.005 m. We denote this success threshold as δ.We report the performance of BiTStar on both δ and δ. Table 1 shows that DiffusionSeeder has a success rate 3x higher and a planning time 3% of that of BiTStar under the less strict success threshold δ. The low success rate of BiTStar provides a benchmark for the difficulty of the motion planning problems under partial observations. BiTStar has a lower jerk compared to DiffusionSeeder as DiffusionSeeder generates faster trajectories, indicated by the lower motion time. When we retimed the trajectories from BiTStar so that trajectories reach the robot's velocity or acceleration limits, the maximum jerk of BiTStar generated trajectories became 81.4 rad/s 3 with a motion time of 1.72s. BiTStar generated trajectories have 21% lower jerk while also being 27% slower than DiffusionSeeder generated trajectories.MπNet [11] is an end-to-end method that takes point clouds and directly predicts waypoints in the configuration space. We use the best pre-trained model (MπNet trained on the Hybrid Expert) as our baseline to compare with. We use a maximum rollout number of 150 for MπNet as in [11] . Additionally, we report the success rate of MπNet under the original success threshold of [11] :δ ′ = (δ t = 0.01m, δ r = 15 • ).Table 1 shows that DiffusionSeeder outperforms MπNet by an order of magnitude in terms of success rate (8.3% vs 85.8%). We hypothesize that this substantial improvement comes from combining the power of a learning approach with a classical motion planner. Since MπNet is a policy that predicts one-step delta joint configuration, it would be significantly slower to combine a multi-step MπNet trajectory (unrolled in a auto-regressive way) with classical methods for reactive control. However, MπNet allows the robot to start moving after the first inference is completed (∼7 ms), while DiffusionSeeder always gives a complete trajectory in 17 ms.cuRobo uses heuristics to generate seed trajectories and its success rate depends heavily on the quality of seed trajectories it uses to optimize. At each attempt, cuRobo samples a new batch of seeds. From Table 1 , the success rate of cuRobo increases monotonically with N atp , indicating the inefficiency of its heuristic sampling as it needs to sample N atp times to achieve a high success rate, while DiffusionSeeder combined with cuRobo achieves higher performance in one attempt. Another effect of seed trajectories is on the number of optimization iterations. We set N iters = 475 for cuRobo to optimize the seed trajectories, which was shown to achieve the highest success rate for cuRobo. DiffusionSeeder can achieve higher performance with significantly fewer iterations (N iters = 25), suggesting the generated seed trajectories are closer to the optimal collision-free trajectories.Both cuRobo and DiffusionSeeder outperform MπNet and BiTStar by a large margin in planning speed. DiffusionSeeder-50 and cuRobo-1 are 11x and 6x faster than MπNet with success threshold δ ′ , respectively, in terms of total time. Since both cuRobo and DiffusionSeeder use nvblox for ESDF reconstruction, we compare the planning time between cuRobo and DiffusionSeeder instead of the total time. On average, DiffusionSeeder-50 is 3x faster than cuRobo-1 with a 30% higher success rate. Compared to best-performing cuRobo-100, DiffusionSeeder-50 is 12x faster with a 10% higher success rate. We show the 75th and 98th quantile of the planning time of cuRobo-100 and DiffusionSeeder-50 in both partially and fully observed environments (Appendix 7.8.6) in Figure 3 , where cuRobo planning time has a great discrepancy across the mean, 75th quantile, and 98th quantile. DiffusionSeeder-50 is 4x faster than cuRobo-100 on the 75th quantile and 36x faster on 98th quantile in partially observed environments. While cuRobo shows high variance in planning time, DiffusionSeeder achieves relatively consistent performance among all scenes, indicating its advantage in generating scene-specific seeds.We conduct experiments on a Franka Panda robot across 6 scenes, categorized into 3 difficulty tiers with 2 scenes from each tier and also an empty scene. Some of these scenes are shown in Figure 4 . None of the environment setups are part of our training dataset. In each scene, we select three poses that the robot needs to reach sequentially, repeating 5 times. A method is considered successful if it avoids collisions across all 5 trials in a scene. In addition to success, we also report the time the robot was executing trajectories as Motion Time. More details are in Appendix 7.9.From Table 2 , DiffusionSeeder failed once among the 7 scenes with an average success rate of 86% while cuRobo failed 3 times with an average success rate of 57%, demonstrating the sim2real transfer of DiffusionSeeder. DiffusionSeeder outperforms cuRobo on planning time with an average planning time of 26ms while cuRobo has a planning time of 65ms. The motion time of the planned trajectories of DiffusionSeeder is higher than that of cuRobo, similar to that in simulation experiments. As discussed in Appendix 7.8.5, we hypothesize this may be attributed to the additional weight α on loss for non-linear trajectories, resulting DiffusionSeeder to generate more non-linear trajectories, which are less likely to collide but also have higher motion time. All failures for cuRobo was due to limited view of the obstacles from a single camera. The one environment where Diffu-sionSeeder failed was because the tree obstacle did not fully fit in the view of the camera. 
conclusion and limitation: 
 We propose DiffusionSeeder, a diffusion-based model for generating initial seed trajectories for motion planning in novel scenes from just depth observations. Integrated with cuRobo, it achieves up to 36× speedup on complex motion-planning tasks. DiffusionSeeder is trained on a large-scale synthetic dataset but shows generalization to unknown real world scenes and observations. By utilizing an environment encoder and a broad set of views of each scene during training, DiffusionSeeder generates collision-free seed trajectories under partial observation. In future, we hope to explore the capabilities of the environment encoder for single-view scene geometry understanding and investigate its applicability in other robotic domains, such as grasping and manipulation.DiffusionSeeder has a few limitations. It generates trajectories based on a fixed external camera view, making it less effective when the goal pose is occluded. We hope to extend DiffusionSeeder to incorporate multiple input views and more diverse camera poses in the future, to benefit from multicamera or wrist-mounted camera setups. DiffusionSeeder is only trained on the Franka robot. We hope to extend DiffusionSeeder to different robots in the future. While we empirically observed that the nonlinearity introduced in Eq. 2 improves the performance, there is a lack of thorough analysis on the effectiveness of Eq. 2 compared to a regular MSE loss. A comprehensive study on the loss function design for robotic motion planning can be an interesting future direction. 
problem generation: 
 To increase data diversity, we sample additional pairs of start and end state poses near the original start and end poses. Since most scenes have obstacles near the end pose, we also sample problems near the end pose to encourage difficult trajectory generation. Specifically, for each scene, we obtain the start pose in Cartesian space from the MπNet start state and sample N-1 new motion planning problems (start and end pose pairs) within a bounding box of 0.3m along the X-Y-and Z-axes around the original pose. We sample N new problems within a bounding box of 0.3m near the end pose. We also sample N pairs of poses in free space, within a cuboid of length 0.6m, to increase our coverage over the joint configuration space of the robot. This results in 3N start and end pose pairs for each scene. We run inverse kinematics for each pose in the problem to filter out ones where either pose is in collision with the scene.We pass each motion planning problem to cuRobo to generate smooth and collision-free trajectories with a fixed trajectory length T = 32. We define the trajectory length as the number of waypoints and perform linear interpolation between the points. Problems for which cuRobo is unable to find a feasible solution are filtered out and discarded. With N = 5, we generated 15M problems with feasible and smooth solutions of size 32×7 on 3M unique scenes.Each camera pose is represented by an azimuth a, an elevation e, and a radius r. For each scene, we discretize orientations to a fixed grid of M = N a • N e (where N a are discretization resolution on azimuth and N e is discretization resolution on elevation) camera poses on a sphere centered at a scene mesh and pointing inward toward the center of the scene. Azimuths range is [-π/4, π/4], elevations range is [π/5, 2π/5], and radius r range is[0.7 × R, 1.1 × R]where R is the radius of scene mesh. We apply uniform random noise ϵ a ∈ [-π/8, π/8] and ϵ e ∈ [-π/15, π/15] to each discretized a, e respectively to obtain diverse camera poses across different scenes.We render depth images from each camera pose and filter out camera poses where the problem end pose is not visible in the rendered depth images. The goal is visible if the projection of the goal in the camera view has larger depth than the goal itself-in other words, none of the observed pixels can occlude the goal in the camera frame. Using N a = 4, N e = 3 and saving up to 4 images per scene (randomly sampled from the valid ones), we rendered 12M depth images of size 256 × 256 on 3M scenes.A DDPM parameterized by θ denoises a given x K for K iterations to generate x 0 , where x 0 ∼ q(x 0 ).x 1:K are generated through a diffusion process q(x k |x k-1 ), modeled as a Markov chain, which gradually adds Gaussian noise to x 0 . The training objective of a DDPM is therefore to minimize the negative log likelihood E[-log p θ (x 0 )], which is upper bounded by [13] E q [-log p θ (x K ) -EQUATIONAs the upper bound is calculated in expectation of the diffusion process q(x k |x k-1 ), we sample different time steps k ∈ [0, K] during the training for estimating the expectation. The denoising process of DDPM is defined as a Markov chain with a learned Gaussian transitions such thatEQUATIONAs shown in [13] , the parameterization of the denoising process can be modified to train a noise prediction model to predict the noise from x k . During training, for each problem, we sample one depth image from the pre-rendered depth images. We also reverse the start and end pose, and the order of the ground truth trajectories half of the time to generate problems going from the end to the start, for further data augmentation. The ground truth trajectories and the input robot start configuration are normalized to be from 0 to 1 by dividing the joint position by the corresponding joint mechanical limit, to improve the training stability. We then denormalize the output of the diffusion model by multiplying with the joint mechanical limit.As the diffusion and denoising process of a DDPM are both Markovian, DDPMs generate x 0 by denoising K steps from x K , resulting in longer inference time. Denoising Diffusion Implicit Models (DDIM) [31] are an efficient class of iterative implicit probabilistic models. They have the same training objective as DDPMs but model non-Markovian diffusion processes. DDIMs thus allow the diffusion model trained with DDPM to generate x k directly from x k+m without the generation of the intermediate steps, speeding up the inference time by greatly reducing the number of denoising iterations. During inference time, we randomly sample a noise vector of the same shape as the trajectory, and use a DDIM to generate a denoised trajectory of shape 32×7. We convert the trained model to BFLOAT16 for accelerated inference and implement the DDIM step as a fused CUDA kernel leveraging JIT scripting in Pytorch 2.0. We encode the full inference step in a single CUDA Graph to further reduce python overheads. These techniques give us a 6ms inference time on an Nvidia GeForce RTX 4090, which includes the inference of the vision encoder, followed by 5 steps of denoising.We use the trajectory optimization problem formulated in cuRobo [2] , which we write briefly below,min τ C goal (X d , F K(q T )) + C smooth (τ ) + C self (τ ) + C world (τ ) s.t. joint limitswhere C goal is the cost for reaching the goal pose at the final timestep T , C smooth is a cost term to encourage smooth minimum-jerk trajectories, C self and C world are self and world collision avoidance cost terms.We consider the following quantitative metrics: We represent scenes with a single depth image. Both cuRobo and DiffusionSeeder use nvblox with the given depth image for collision checking. As we notice the performance of nvblox is sensitive to the image size, we render depth images of size 640 × 640 at the test time for nvblox EDSF reconstruction and downsample them to 256 × 256 as the input to the diffusion model.To evaluate the generalization ability of DiffusionSeeder, we evaluate DiffusionSeeder on narrow passage problems. We generate 500 problems on 50 scenes consisting of 10 cuboid obstacles from the sparrow dataset [35] . Results are shown in Table 3 . Both DiffusionSeeder and cuRobo achieve a success rate of 52.4% when given partial observation of depth images. Examples of the problems and the depth observations are shown in Figure 6 . As DiffusionSeeder is never trained on such scenes, the performance of DiffusionSeeder is similar to cuRobo. In the future work, we expect to include more diverse scene types in the training of DiffusionSeeder. used. From the results, the gradient guidance improves the performance marginally. We suspect that the improvement is marginal because DiffusionSeeder is conditioned on the scene and can already output seed trajectories at the low cost region of the scene. We leave how to better incorporate the gradient guidance in the diffusion denoising process to future work.From Table 1 , DiffusionSeeder-475 has the highest success rate of 86.2% and DiffusionSeeder-25 has the lowest planning time of 15ms, with a success rate of 85.1%. The trajectory quality increases with N iters , indicated by the decreasing jerk, motion time, pose translation, and quaternion error, at the cost of increasing planning time from 15ms to 45ms. We observe that the success rate doesn't vary significantly with N iters . We hypothesize that this is because the diffusion model is trained on the trajectories optimized for motion planning cost on the ground truth mesh and additionally sees multiple different views of the scene during training. It implicitly minimizes the trajectory optimization cost and can do implicit scene completion from partial observations. Therefore, the seed trajectories are closer to the optimal collision-free trajectory while avoiding collision in unobservable areas and require fewer optimization steps to generate collision-free trajectories. Contrarily, cuRobo optimizes on trajectories generated by a graph-based planner or linear interpolation that are less optimal, requiring more optimization iterations to minimize the motion optimization cost.Among all methods, BiTStar has the lowest jerk but also the lowest success rate. MπNet has the highest motion time and target pose tracking error. The maximum jerk of DiffusionSeeder generated trajectories is slightly higher than cuRobo but still far below the robot's mechanical limit. DiffusionSeeder has slightly higher motion time, likely coming from the non-linear trajectories that cuRobo fails to solve. The additional weight α on loss for non-linear trajectories (see Section 4.3) may also contribute to this high motion time of DiffusionSeeder. Both DiffusionSeeder and cuRobo have sub-millimeter target pose translation tracking error and low quaternion tracking error. In this set of experiments, we pass the ground truth mesh for optimizer collision checking for BiT-Star, cuRobo, and DiffusionSeeder. For MπNet, we pass point clouds sampled from the ground truth mesh as in MπNet. We calculate each metric for successful trajectories, with the average value summarized in Table 5 . From Table 5 , BiTStar has a success rate of 61% under success threshold δ, providing a benchmark of the difficulty of the problems. cuRobo-100 has the highest success rate of 99.7%, while DiffusionSeeder-50 has a success rate of 97%. It's worth noting that the diffusion model of DiffusionSeeder still takes partial observations of a depth image while cuRobo takes in full observations, which explains the slightly lower success rate of DiffusionSeeder compared to cuRobo and the consistent success rate across different N iters . The trajectory quality with N iters shows similar trends to those in partially observed environments. In fully observed environments, cuRobo-100 is still significantly slower than DiffusionSeeder as shown in Figure 3b , where DiffusionSeeder-50 is 6x faster on average, 4x faster on 75th quantile and 22x faster on the 98th quantile. As aforementioned, cuRobo calls a graph based planner when the linear interpolation trajectory fails for complicated problems, resulting in the significant slow down on obstacle-dense scenes. In contrast, DiffusionSeeder has a consistent planning time over all scene types.We study the performance of DiffusionSeeder with N trajs in partially observed environments. We set N iters = 200. Results are shown in Table 6 . The success rate of DiffusionSeeder increases monotonically upto N trajs = 12, suggesting N trajs = 12 is able to represent the potential modalities sufficiently. The success rate of N trajs = 12 is 1.99x higher than that of N trajs = 1, highlighting the benefits of using the diffusion model to generate multiple diverse seeds, compared to motion planning policies that generate one initial trajectory through multi-step rollout. We notice that the success rate is not increasing monotonically after N trajs = 12. We hypothesize this might be because as N trajs increases, the ratio of linear trajectories also increases, resulting the optimizer to optimize a more linear trajectory, which is implied by the lower motion time for higher N trajs .N We evaluate the performance of DiffusionSeeder without integration of cuRobo (i.e., no trajectory optimization). We sample N trajs from the diffusion model and choose the trajectory that has the lowest translation error as the final solution. We do collision checking with the ground truth mesh and check if the translation error and quaternion error is below the success threshold to calculate the success rate.Results are summarized in Table 8 . As N trajs increases, the success rate increases monotonically. The collision-free rate, which is consistent with respect to the N trajs , is the upper limit of the success rate of the diffusion model as collision-free trajectories can have large end pose errors. The success rate of the diffusion model is much higher than MπNet (Table 1 ), indicating that the diffusion model is able to generate environment dependent seeds. However, the success rate is much lower than that of DiffusionSeeder integrated with cuRobo, highlighting the benefits of optimizing the diffusion seeds with explicit collision checking using nvblox and other cost terms. In addition, diffusion generated trajectories tend to have a high maximum jerk of around 270 rad/s 3 , which is significantly reduced by cuRobo optimization to around 106 rad/s 3 .We use a Realsense D435 depth camera mounted opposite to the robot to obtain depth images. The robot is segmented and removed from the depth images and then sent to DiffusionSeeder and nvblox. None of the environment setups are part of our training dataset, in addition the chosen camera view causes severe occlusions in the environment when obstacles are present as seen in Figure 1 . Hence we used N iters = 100 for DiffusionSeeder as using 50 did not get us high-quality solutions. For cuRobo, we use the default number of iterations (475) provided in their code base and have N atp = 10. For both methods, we run the Franka Panda with a joint impedance controller so that collisions in the world do not damage the robot. In addition, we set the maximum allowed velocity to 65% of the robot's limits across the methods to avoid any catastrophic damage due to occlusions in the world. 
 
 Appendix: Success Rate Jerk (rad/s 3 ) Motion Time(s) T Err(mm) Q Err( • Success Rate: the percentage of collision-free trajectories with the end pose translation error and quaternion error lower than δ t and δ r respectively.• Plan Time: the time for the planner to generate the trajectories given the observation.• Jerk: the maximum jerk of the flattened trajectories over all time steps and robot joints.• Translation Error: the mean squared error between the achieved end pose position and the target pose position.• Quaternion Error: the cosine distance between the achieved end pose quaternion and desired end pose quaternion.• Motion Time: overall execution time of the trajectories accounting for a robot's mechanical limits, such as maximum velocity, acceleration, and jerk. 


Decision: The paper proposes the use of the diffusion method to generate conditional initial trajectories for trajectory optimization. Overall, the paper is well-written and easy to follow. However, it is necessary to demonstrate the proposed work in more challenging narrow passages to validate its capability. Furthermore, a comparison against relevant baselines such as DiffusionSeede is necessary to validate the contribution of the proposed approach over prior work. Additionally, the results should be clearly explained, including why the proposed method has more jerks in the trajectories, and the design of the objective function also needs to be motivated.

Update: The revised version has addressed the reviewer's comments to a good extent.

Reviews: review: title: title: DiffusionSeeder: Seeding Motion Optimization with Diffusion for Rapid Motion Planning
paper_summary: paper_summary: The paper proposes diffusion model-based trajectory generation, which can be used to improve planning efficiency and the success rate of trajectory optimization in complex environments. The method additionally addresses handling partial observability by considering environment conditioning with a depth image obtained by a fixed camera. The proposed method shows promising results compared to various baselines in both planning time and success rate.
main_review: main_review: Strengths:
- The overall presentation is clear, easy to read, and understand.
- The motivation of the work is convincing as it is closely connected with cuRobo.
- The proposed idea is novel, using the output of diffusion models as good-quality seed trajectories to increase the likelihood of finding a solution trajectory.
- The literature review appears comprehensive, as it covers both classical methods and recent learning-based methods.
- The experimental results are promising, and the evaluations are exhaustive. They include various baselines, and diverse aspects have been validated.

Weaknesses:
- Occlusion evaluation is not clearly demonstrated, making the approach to dealing with partial observability questionable. Since this is one of the major contributions, this aspect must be systematically evaluated. How severe occlusion can the method handle?
- A new loss function (Equation (2)) is proposed, but its effectiveness is not evaluated. Therefore, it is unclear why the new loss function is more beneficial than the original one.
questions: questions: - Please address the two comments in the weaknesses above.
- In Figure 1’s caption, “nvblox [1] online from the depth images …” What does "online" mean here? I thought the overall pipeline computes a trajectory offline, so this part was confusing. 
- Figure 2 does not include the “time step” latent vector.
- In Section 5.1: 575 dim -> 576 dim
- At the end of Section 5.1: “831-dimensional conditional vector for each layer.” Which layer does this refer to?
- In Figure 4, mark the goals. Otherwise, it’s hard to judge whether the problem is easy or hard.
limitations: limitations: 3
review_summary: review_summary: The paper's contribution appears significant, and the presentation and technical details are clear and sufficient.
title: title: Good work with broad experiments
paper_summary: paper_summary: This paper propose to use diffusion method to generate conditional initial trajectories for trajectory optimization.
main_review: main_review: This work proposes to train a diffusion model for a large-scale motion planning problem. The output paths of the model become the initial trajectories for a trajectory optimizer. 

The method supports the raw depth image input and utilizes the strength of ViT. The diffusion part uses the DDPM model. 

The paper's method is not very novel while the experiments are solid. The paper demonstrates it is possible to make an end-to-end motion planning from raw observation. 

However, despite the variance of the input observation, the generated paths have some similar patterns, for example, the manipulator's end effector goes up and down. I think it reflects the planning problems' similarity in the dataset. While real-world problems may include more cases, e.g. the narrow passage problem in Riemannian Motion Policies [2].

Reference missing. There exists a concurrent diffusion work [1] for motion planning on MpiNet's dataset. 

[1] EDMP: Ensemble-of-costs-guided Diffusion for Motion Planning

[2] Riemannian Motion Policies
questions: questions: 1. I noticed the experiments in the video are either the end effector going up and going down, or the end effector going into the cabinet. Could this method work for more challenging narrow passage problems? For example, multiple links go into the cabinet.
limitations: limitations: 3
review_summary: review_summary: This work propose to use ViT conditioned difffusion to generate initial planning paths and demonstrates its effectiveness on a large dataset. The method can have a good impact for further large scaled robot learning.
title: title: Essentially a big data approach.
paper_summary: paper_summary: DiffusionSeeder uses a conditional Denoising Diffusion Probabilistic Model (DDPM) to generate diverse, high-quality seed trajectories from initial depth images of the environment. These seed trajectories are then optimized using cuRobo, a GPU-accelerated motion optimizer. The key claim is that DiffusionSeeder accelerates motion planning and increases success rates in complex environments with image observations by providing high-quality seed trajectories for trajectory optimization, trained on big datasets. The evaluation involves both simulated and real-world experiments using the Franka robot. In simulations, DiffusionSeeder is tested on 3M scenes from the MπNet dataset, comparing performance with BiTStar, MπNet, and cuRobo.
main_review: main_review: **Novelty & Significance**

1. The method is essentially an engineering work that vision-conditional diffusion models initialize downstream trajectory optimizers, e.g., curobo optimizers. This is essentially without the prior and likelihood blending in [1]. The contribution is, therefore, judged by the significance of the experiment results.

2. The loss Eq. 2 needs proper motivations since matching end-effector poses only basically discard self-collision and collision information from the planner data. Moreover, FK is inherently highly non-linear; matching poses noise via FK amplifies noises. Perhaps it works due to having an enormous dataset, but I cannot see why this helps training diffusion.

3. Although diversity is not a focus of this work, I believe that with such a big dataset, the diffusion prior can generate very diverse and high-quality modes for downstream trajectory optimization.

**Quality & Clarity**

1. The method descriptions and figures are well-written and easy to read.

2. The experiment outline has good and diverse test environments. The baselines are also adequate. However, since this work is very close to [1] (although [1] does not condition on image/depth observations), this work should still compare to [1] by reusing the trained diffusion prior and curobo [2] planner with the blending in Algorithm 1 of [1]. This can be considered an ablation study of with vs. without blending diffusion prior update and likelihood (i.e., planner) gradient update.

-------------------------
**Post Rebuttal Comments**

I would like to thank the authors for their efforts in addressing my concerns. I raise my score to Weak Accept.


**References**

[1] Carvalho, Joao, et al. "Motion planning diffusion: Learning and planning of robot motions with diffusion models." 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023.

[2] Sundaralingam, Balakumar, et al. "Curobo: Parallelized collision-free robot motion generation." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.
questions: questions: 1. Why does DiffusionSeed have high jerks compared to a sampling-based approach such as BitStar?
2. Please compare DiffusionSeeder to [1], with the focus on comparing two paradigms of coupling vs. decoupling diffusion prior update and likelihood (i.e., planner) gradient update, to see if the best possible method configuration of diffusion prior for motion planning.
limitations: limitations: 3
review_summary: review_summary: This works has clear motivations and clear engineering description. However, I am not totally convinced on the experiment statistics. I will increase my score when additional experiment results added after the rebuttal.