DiffusionSeeder: Seeding Motion Optimization with Diffusion for Rapid Motion Planning
Authors: Huang Huang
Balakumar Sundaralingam
Arsalan Mousavian
Adithyavairavan Murali
Ken Goldberg
Dieter Fox
Venue: CoRL 2024 (2024)
Keywords: Robot Motion Planning
Diffusion Model
TL;DR: We propose DiffusionSeeder, a diffusion based approach that generates trajectories to seed motion optimization for rapid robot motion planning.

Abstract: Running optimization across many parallel seeds leveraging GPU compute [2] have relaxed the need for a good initialization, but this can fail if the problem is highly non-convex as all seeds could get stuck in local minima. One such setting is collision-free motion optimization for robot manipulation, where optimization converges quickly on easy problems but struggle in obstacle dense environments (e.g., a cluttered cabinet or table). In these situations, graph based planning algorithms are called to obtain seeds, resulting significant slowdowns. We propose DiffusionSeeder, a diffusion based approach that generates trajectories to seed motion optimization for rapid robot motion planning. DiffusionSeeder takes the initial depth image observation of the scene and generates high quality, multi-modal trajectories that are then fine-tuned with few iterations of motion optimization. We integrated DiffusionSeeder with cuRobo, a GPU-accelerated motion optimization method, to generate the seed trajectories which results in 12x speed up on average, and 36x speed up for more complicated problems, while achieving 10% higher success rate in partially observed simulation environments. Our results prove the effectiveness of using diverse solutions from learned diffusion model. Physical experiments on a Franka robot demonstrate the sim2real transfer of DiffusionSeeder to the real robot, with an average success rate of 86% and planning time of 26ms, increasing on cuRobo by 51% higher success rate and 2.5x speed up. The code and the model weights will be available after publication.

DiffusionSeeder: Seeding Motion Optimization with Diffusion for Rapid Motion Planning
Huang Huang UC Berkeley
Balakumar Sundaralingam
Arsalan Mousavian Nvidia
Nvidia
Adithyavairavan Murali Nvidia
Ken Goldberg UC Berkeley
Dieter Fox
Nvidia
(A) Observations and Inputs
(C) Final Trajectory
Depth Observation
Nvblox Voxel
Start State Target pose
(B) DiffusionSeeder Trajectories
propose DiffusionSeeder, diffusion based method that generates diverse, multi-modal trajec- tories to warm start motion optimization, greatly reducing the planning time on "hard prohlems" ((A) The model takes depth images, the robot start configuration, and target pose. (B) DiffusionSeeder generates di- multi-modal trajectories. where tance Field (ESDF) using avblox [1] from depth images used by motion optimizer cuRobo to move trajectorics out of collision. (C) The final output from the motion optimizer smooth, fast, and collision- free.
Figure I:
trajectories could be in collision.A dynamic Euclidean Signed Dis-
Abstract: Running optimization across many parallel seeds leveraging GPU com- pute [2] have relaxed the need for good initialization, but this can fail if the problem is highly non-convex as all seeds could get stuck in local minima. One such setting collision-free motion optimization for robot manipulation, where optimization converges quickly easy problems but struggle in obstacle dense environments (e.g., cluttered cabinet or table). In these situations, graph-based planning algorithms are used to obtain seeds, resulting in significant slowdowns. We propose DiffusionSeeder, a diffusion based approach that generates trajecto- ries to seed motion optimization for rapid robot motion planning. DiffusionSeeder takes the initial depth image observation of the scene and generates high quality, multi-modal trajectories that are then fine-tuned with few iterations of motion optimization. We integrate DiffusionSeeder generate the seed trajectories for cuRobo, GPU-accelerated motion optimization method, which results in 12x speed up average, and 36x speed up for more complicated problems, while achieving 10% higher success rate in partially observed simulation environments. Our results show the effectiveness of using diverse solutions from a learned dif- fusion model. Physical experiments on a Franka robot demonstrate the sim2real transfer of DiffusionSeeder to the real robot, with an average success rate of 86% and planning time of 26ms, improving on cuRobo by 51% higher success rate while also being 2.5x faster. More information are in the website.
Keywords: Robot Motion Planning, Diffusion Model
Work done during internship at NVIDIA
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
Robot motion generation aims to find collision-free path in robot configuration space to reach a desired goal pose from a starting configuration. It is challenging due to the high dimensional robot configuration space, where each configuration corresponds a combination of robot joint space and an environment representation. This problem has been studied using sampling-basee ppased methods [3, 4), which sample configurations in the configuration space and checks for collisions in-between until complete path is found. Others have leveraged optimization-base methods [5, 6. 7J. which optimize an initial path under specific objectives and constraints, to solve this problem. For both approaches, planning time increases significantly with the complexity of the configuration space [8], resulting in a large variance in planning time for different problems. Furthermore, sampling-basee baand methods of- methods, which can generate smooth paths directly, are highly sensitive to the initial seed. Several optimization- based algorithms [2, 9, 5] address this problem by starting from set of seed trajectories for opti- mization. The quality of seed trajectories are crucial, as an absence of good seeds can lead to failure in densely populated environments. Therefore, modern implementations of planners are forced to resort to graph-based planning algorithms to guarantee completeness at the cost of speed.
ten require post-processing to smooth the generated paths, while optimization-
Recently, learning-basee based methods have demonstrated great promise motion generation without access to ground-truth state, achieving faster planning than sampling-based methods [9, 10, 11]. Some work [8, 12, 11] predict the next joint state iteratively instead of the entire trajectory, increasing the planning time. Learning-based methods often lack theoretical guarantees, fail to generalize to out of distribution scenes, and tend to generate sub-optimal plans as we later show in our results.
We aim to develop a rapid and practical off-the-shelf motion planning pipeline that works with partial observations. We propose DiffusionSeeder, which generates diverse, high-quality paths to seed the optimization based motion planner, shown in Figure significantly reducing both the seed trajectory generation time and the number of optimization steps required. Denoising Diffusion Probabilistic Models (DDPM) [13], a class of generative networks that model the output genera- tion as a denoising process, have shown success in capturing complex multi-modal distributions in image generation and, recently, in robot manipulation [14]. Robot motion planning is well known to be a multimmdal problem, making DDPM a great fit for seed trajectories generation. Diffu- sionSeeder uses a conditional DDPM, consisting of an observation encoder and a noise prediction network for seed trajectory generation. We represent the partially observed environment with depth observations, which faster to process than point clouds [11, 8, 12]. We train DiffusionSeeder on 3M simulated scenes from MrNet [11]. We use cuRobo, a fast motion optimizer utilizing GPU prprreparllllizzaiinn to optimize DiffusionSeeder generated seeds for final trajectories.
This paper makes the following contributions:
DiffusionSeeder, conditional DDPM that generates high-quality seed trajectories within milliseconds from a raw depth image, camera pose, start joint configuration and end pose.
robot motion planning dataset consisting of 15M smooth and fast trajectories generated by cuRobo on 3M M*Net [11] scenes in simulation.
Integration of DiffusionSeeder with cuRobo, achieving 12x average speed up over standard cuRobo (and 36x speed up 98th quantile), while achieving 10% higher success rate on partially occluded views of scene.
2 Related Work
Motion planning for robotics has been broadly explored. Search-based planning methods such as A [15, 16] are complete, fast, and optimal for problems with discrete state spaces. Sampling- methods such as RRT [3], and PRM [4] are asymptotically optimal but can generate sub-optimal and erratic trajectories in practice with limited compute budget. Optimization- methods [5, 7] susceptible to local minima in complex problem spaces. GPU-parallelized computations have
been leveraged to address this local minima issue [2]. cuRobo [2] is GPU-accelerated motion

---

optimizer that solves on many seeds with linear interpolations from start to goal configurations in parallel to avoid local minima. In the case of failure, it falls back to a graph based motion planner to generate new seeds which significantly slows down the planning time, taking up to 0.5 seconds. Our work aims to mitigate this issue by using a diffusion model to generate high quality multi-modal seed trajectories, resulting in fast convergence and removing the need for graph based planner.
Many works have utilized a learned neural network for generating pepironment-dependeen sampling distributions [17, 9, 10, 18, 19], accelerating sampling-based planning by biasing samples toward collision-free joint configurations. But the planning time still scales with the environment com- plexity. Other approaches learn a collision model [20, 21] for fast collision checks, but still rely on exhaustive sampling with Model Predictive Path Integral (MPPI) [22. 23] and local controller. Other works 811,8, 12] have formulated the motion planner a policy which is queried sequentially to generate trajectory. Motion Planning Networks (MPNet) [8, 12] and Net [11] learn policies to directly predict the next joint configuration from point cloud. However, motion planning poli- cies require multiple rollout steps to generating one trajectory, resulting in longer planning time. There is no further optimization on the generated trajectory, which can result in suboptimal trajecto- ries. Some approaches, such as Ichnowski et al. [24], warm-start optimizers with ketwork-generatd seeds, but lack generalization to new scenes and output only one trajectory. Yoon et al. [25] learn reinforcement learning policy to generate an initial trajectory, but multi-step rollouts are slow and limited to one result, while DiffusionSeeder can generate multiple diverse trajectories quickly.
Diffusion models have been leveraged for trajectory optimization. Janner et al. [26] trains diffusion model per environment, making it impractical for diverse scenes. Carvalho et al. [27] and Saha et al. [28] train an observation/emviiommnnt agnostic diffusion model and leverage the problem costs to guide the diffusion sampling to generate trajectories priors satisfying the scene-specific constraints, which may struggle to generate collision-free trajectories in partially observed cluttered environ- ments. DiffusionSeeder is conditioned specific trajectories for various environments, which are more efficient for motion planning in com- plicated environments. Huang et al. [29] represents the environment as point clouds and utilizes goal-oriented planning for robot motion planning, formulated as a motion in-painting problem, which requires multiple policy rollout steps for generating one trajectory. DiffusionSeeder takes in depth observations as the input to a Vision Transformer (ViT), contrast to slower-to-process point clouds. Further, DiffusionSeeder generates multiple full trajectories for each inference pass of the diffusion model, avoiding multi-step policy rollouts.
environment encoding and can generate environment-
Problem Statement
We study the robotics motion planning problem. The goal is to generate a fast, smooth and collision- free trajectory to bring robot from starting configuration to a goal pose, given depth image of dimension H x W, the calibrated intrinsic and extrinsic matrices of third-person view camera, the starting robot joint state, qu, and the goal robot end effector pose (a 7d vector of translation and quaternion in Cartesian space). We aim to develop a seeder that can generate diverse and high quality seed trajectories to accelerate a motion optimizer to rapidly generate a collision free trajectory 7 with position and rotation errors below given thresholds (6, and ö,, respectively). while minimizing the jerk and motion time of the generated trajectory. We define 7 be {(t,, q:)li E [0, T]}, where t, is the time step, qi is the robot joint state at time step t,, and T is the length of the trajectory.
4 Method
4.1 Dataset Generation
We generate our dataset using cuRobo for a 7-DoF Franka robot on 3M MrNNet training scenes [11] in three categories: cubby, tabletop and dresser (Appendix Fig 5). A motion planning problem in the Mreet dataset consists of a scene mesh geometry, start and end pose. To increase data diversity, sample additional pairs of start and end poses based on the original start and end poses. Problems for which cuRobo is unable to find feasible solution are discarded. We generated 15M problems
3

---

with feasible and smooth solutions of size 32x7 7 on 3M scenes, where 32 is the trajectory length T and 7 is the robot DoF. More details are in Appendix 7.1.
To provide partial observations to the planner, we render depth images for each scene using fixed virtual camera located within the scene. To generalize different view points, we sample 12 dif- ferent camera poses for each scene. We render depth images from each camera pose and filter out camera poses where the problem end pose is not visible in the rendered depth images, resulting 12M rendered depth images of size 256 x 256 on 3M scenes. We save the corresponding camera intrinsic and extrinsic in robot base frame for each depth image. More details are in Appendix 7.2.
4.2 Model Architecture
DiffusionSeeder uses a conditional DDPM, consisting of an environment observation encoder and a conditional noise prediction network, to generate diverse seed trajectories, which can be optimized by an optimizer cuRobo for the final trajectory, as shown in Figure 2.
The observation encoder processes depth images, camera poses, the start joint configuration, and the goal pose to create a single embedding vector as the condition of the denoising diffusion process. Specifically, we use a ViT with 12 layers and 12 heads as the network backbone. Depth images, projected to 3D using camera intrinsics, are divided into 64 x patches where each patch is encoded first and concatenated with the linear projection of the SE(3) homogeneous transformation of each camera pose. We add a positional embedding each patch embedding, resulting single 512- dimensional visual embedding using the class token of VIT. The visual embedding is concatenated with the 64-dimensional encoding vector of the start configuration and the goal pose, resulting in 76-diimensional environment embedding vector for the conditional noise prediction model.
We adopt the CNN based architecture from [14] as the conditional noise prediction model, which 3-level UNet [30] architecture consisting of conditional residual blocks with channel dimensions [256, 512, 1024]. The model encodes the time step into latent vector of 256 dimensions through multi-layer perceptron (MLP), which is concatenated with the environment embedding from the observation encoder, resulting an 831 331-dimensional conditional vector.
4.3 Model Training and Inference
We jointly train the observation encoder and noise prediction model of the DDPM. During training, for each problem, we sample one depth image from the pre-rendered depth images. At each training iteration, we sample ground truth trajectory from the dataset. We randomly select a denoising step E [0, K] and random noise c, which is added to the ground truth trajectories, resulting a noisy trajectory 8 + c. The objective of the noise prediction network @ is to predict the noise added to the original trajectory, with the training loss defined as
L MSE(E, O ((O)))
where the observation encoder and the observation.
In robot motion planning problems, the error in each joint have different impact on the end effector error, due to the non-linearity of the robot forward kinematics (FK) model. To mitigate this issue, we pass both and the predicted noise é to the robot FK model to reflect this non- non-lineane lineariatttt. We use FK to obtain the position of many points sampled on the robot's geometry across all links. We calculate the distance between the predicted positions of these points and the labels (using FK on joint state) as the loss. This loss gives a more direct representation of the error in the Cartesian Space for the whole robot. In addition, as many of the motion planning problems can be solved through linear interpolation solution, we upweight the loss for non-linear solution to encourage the model pay attention to the non-linear solutions, which are often harder to solve. As cuRobo will only call a graph based planner when the linear interpolation solution has failed, we use this to approximate the non-linearity of the trajectories. Together, we define our training loss as
L = (L (.))))
(2)

---

Observation Encoder
Senpled hoka
Final Trajectories
Caniwa Eutinad
MIP
Time
Conditional
Dnvicemnt
Diffusion Model
VIT
Posiin Broading
Optmizor
Depth
Target Powe
Figure 2: DiffusionSeeder trajectory generation pipeline. DiffusionSeeder consists of a VIT observation encoder (blue) and conditional diffusion model (coral). The robot start configuration and target pose are embedded through an MI.P into "problem encoding", which concatenated with the environment embedding. The diffusion model conditions on the concatenated vector and a randomly-samee sampled noise tensor size x2x7 to generate initial trajectories for the optimizer (green). The optimizer takes in the ESDF for collision checking. The final trajectory after optimization is shown on the right.
where 1 (T) is an indicator of whether the ground truth trajectory T is generated from graph based planner and a is a scale factor. We train our model with depth images of size 256 x 256, K = 100, a = and batch size of 256 for 72 epochs, which takes days on NVIDIA A100 GPUs.
We use Denoising Diffusion Implicit Models (DDIM) [31] for faster inference. Combined with CUDA optimization, we achieve 6ms inference time Nvidia GeForce RTX 4090, which includes the inference of the vision encoder and 5 steps of denoising.
4.4 Optimization
We incorporate the trained DiffusionSeeder with an existing trajectory optimizer available in cuRobo. Joint space trajectories from DiffusionSeeder, the robot start configuration qu, target end pose X2 are passed the optimizer optimize for Niers iterations. Details in Appendix 7.6.
In partially observed scenes, for optimizer collision checking, we construct a Euclidean Signed Dis- lance Field (ESDF) using nvblox [1] from input depth images. Nvblox is GPU-accelerated signed distance field construction library designed for robotic path planning, which uses GPU parallel com- putation to efficiently perform volumetric mapping of the world, while also enabling sharing of the generated map in a zero-copy mode with cuRobo for trajectory optimization on the GPU.
Metric
SiTStar
MaNet
cuRoba v0.6.2
DefeeeionSeeder
Condition
Nata
Nirosa
Plan Time (5)
0.32
0.48
0.30
1.95
0.049
0.082
0.207
0.015
0.017
0.020
0.027
0.015
Total Time
0.69
0.65
0.50
1.95
0,079
0.112
0.237
0.045
0.047
0.050
0.057
0.075
Success Rate
26.5%
6.0%
27.4%
8.3%
66.2%
77.7%
77.9%
83.1%
33.8%
84.9%
85.1%
86.2%
Jark (rod//**)
(81.4)
49.9
56.8
60.6
98.5
96.7
97.8
108.8
103.6
99.3
93.5
89.6
Motion Time (S)
1.84 (1.72)
1.98
7.71
1.14
1.18
1.27
Translation Eec (mu)
3.89
4.05
8.66
3.92
0.05
0.06
0.06
0.98
0.95
0.91
0.50
0.78
Quaternion Err
13.3
1.10
7.27
2.68
0.63
0.90
0.93
1.78
1.44
1.20
1.03
0.92
Table 1: Evaluation with partial observation of depth image. Mean values of each metric on successfully solved problems over the 1791 test problems are reported. DiffusionSeeder-50 has similar success DiffusionSSeddr-475 but is 60% faster at planning, we use DiffusionSeeder when reporting primary results. The jerk and motion time of re-timed BitStar trajectories are shown in the parenthesis.
5 Experimental Evaluation
We conduct experiments both in simulation and on physical Franka Panda robot evaluate Dif- fusionSeeder. In the following sections, denote DiffusionSeeder DiffusionSeeder combined with cuRobo, unless otherwise specified. We use the same set of parameters for both simulation and real-world experiments. We run DiffusionSeeder with NNemoosing denoising steps to generate trajec- tories of length T = =32 in the joint space. For each problem, we pass Nunjs randomly sampled noise with the same condition from the observation encoder to the diffusion model to generate Nuajs initial trajectories for optimization. Naemoising = 5, Nuajs = 12 achieves good trade off between generation quality, diversity, and inference time. We consider the following quantitative metrics: success rate, plan time, jerk, translation error &g. quaternion error o,, and motion time (See Appendix 7.7). All metrics are computed over successful trajectories.
5

---

cuRobo
DiffusionSeeder
cuRobo
DifeusionSeeder
1,000
242
Planning Time (ms)
Planning Time (ms)
200
500
100
207
68
59
17
64
16
25
16
17
25
0
Mean
75th Quantile 98th Quantile
Mean
75th Quantile 98th Quantile
(a) Planning in partial observed environments.
(b) Planning fully observed environments.
Figure 3: Planning time for cuRobo- 100 and DiffusionnSeeer-500 in partially and fully observed environment on the MwNet test set [11]
5.1 Simulation Experiments
We evaluate DiffusionSeeder on the MrNet simulation test set of 1791 problems. Each problem has scene from the same scene types the training data but with different configurations (Figure 5). We compare DiffusionSeeder to BiTStarr MrNet and cuRobo(vO.6.2), We use 6, 0.005m and 6, 2.86 as the success threshold for both cuRobo and DiffusionSeeder as in [2].
We represent scenes with a single depth image. Both cuRobo and DiffusionSeeder use nvblox with the given depth image for collision checking, which takes 30ms reconstruct the ESDF. The total time of DiffusionSeeder and cuRobo is the sum of the planning and nvblox reconstruction time. In practice, this can be expedited by rumning nvblox construction in parallel with the diffusion model inference. For BiTSttar mesh generation from the depth image takes 0.17s. For MrNet, we project depth images to point clouds using the camera intrinsics. We evaluate the generated trajectories from all method using the ground truth mesh for collision checking (see Appendix 7.8).
We run cuRobo for maximum of Nap attempts, denoted as cuRobo-Narp- For each attempt, we check if the trajectory meets the success criterion, repeating until successful trajectory is generated or the attempt limit is reached. We use Nap 1, 10, 100 more attempts does not further increase the performance. For DiffusionSeeder, we run Niters optimization iterations for the diffusion gener- ated trajectories, denoted as DiffusionSSeeer--Nine optimization iterations doesn't 't increase the trajectory quality much but increases the planning time. We evaluate methods on Nvidia GeForce RTX 4090.
We use Niters 25, 50, 100, 200, 475, as more
The average values of each metric are summarized in Table 1. Experiments in fully observed en- vironments using ground-truth meshes, ablations on Nijjs and number of depth images, Diffusion- Seeder only without cuRobo and more discussions are included in Appendix 7.8.
5.1.1 DiffusionSeeder vs Sampling Based Planner
BiTStar [32] a sampling- based planner that utilizes the benefits of the random geometric graph. can generate high quality and optimal solutions quickly compared other sampling-base based methods, especially in high dimensional problems, and has the properties of probabilistic completeness and asymptotic optimality. We incorporate BiTStar from OMPI. [33] into Movelt2 [34] with a timeout of 5s and a maximum of 10 attempts per timeout. Since MoveIt2 constructs goal constraints using the Euler angle error on each axis, we set the angle error tolerance on goal pose to be 0.01 rad for each axis and the translation error tolerance to be 0.005 m. We denote this success threshold as 8. We report the performance Bit'tar on both å and 5. Table 1 shows that DiffusionSeeder has success rate 3x higher and a planning time 3% of that of BiTStar under the less strict success threshold 8. The low success rate of BiTStar provides a benchmark for the difficulty of the motion planning problems under partial observations. BitStar has a lower jerk compared to DiffusionSeeder

---

(a) Easy
(b) Hard
(c.1) Medium: cuRobo
(c.2) Medium: D.S.
Figure 4: Physical experiments scenes: (a) an easy scene with a box, (b) a hard scene where both cuRobo and DiffusionSeedor failed, (c) a medium seene. (c.1) shows cuRobo colliding while (c.2) shows DiffusionSeedor avoiding collision in the same scene. Thene.The start and end position are marked by the red stars. The two positions are to the left and right of the obstacles, requiring the robot to navigate around the obstacle for safe exccution.
as DiffusionSeeder generates faster trajectories, indicated by the lower motion time. When we re- timed the trajectories from BiTStar so that trajectories reach the robot's velocity or acceleration limits, the maximum jerk of BiTStar generated trajectories became 81 rad/s* 8 with a motion time of 1.72s. BiTStar generated trajectories have 21% % lower jerk while also being 27% slower than DiffusionSeeder generated trajectories.
5.1.2 DiffusionSeeder vs Prior End-to-End Approach
M*Net [I1] is end-to-end method that takes point clouds and directly predicts waypoints in the configuration space. We use the best pre-trained model (MrNet trained on the Hybrid Expert) as our baseline to compare with. We use a maximum rollout number of 150 for MrNet as in [11]. Additionally, we report the success rate of MrNet under the original success threshold of [11]: = (6; = 0.01m, 6, 15"). Table shows that DiffusionSeeder outperforms MrNet by an order of magnitude in terms of success rate (8.3% vs 85.8%). We hypothesize that this substantial im- provement comes from combining the power of learning approach with classical motion planner. Since MrNet is policy that predicts one-step delta joint configuration, it would be significantly slower to combine multi-step MrNet trajectory (unrolled in rugo-regressive way) with classi- cal methods for reactive control. However, MrNet allows the robot to start moving after the first inference is completed (~~7 ms), while DiffusionSeeder always gives a complete trajectory in 17 ms.
5.1.3 DiffusionSeeder Heuristic Seed Generation
cuRobo uses heuristics to generate seed trajectories and its success rate depends heavily on the quality of seed trajectories uses to optimize. At each attempt, cuRobo samples new batch of seeds. From Table 1, the success rate of cuRobo increases monotonically with Nap- indicating the inefficiency of its heuristic sampling while DiffusionSeeder combined with cuRobo achieves higher performance in one attempt. Another effect of seed trajectories is on the number of optimization iterations. We set Niss 475 for cuRobo to optimize the seed trajectories, which was shown to achieve the highest success rate for cuRobo. DiffusionSeeder can achieve higher performance with significantly fewer iterations (Nuers = 25), suggesting the generated seed trajectories are closer to the optimal collision-free trajectories.
needs sample Naup times to achieve high success rate,
5.1.4 Planning Time Comparisons
Both cuRobo and DiffusionSeeder outperform MrNet and BiTStar by large margin in planning speed. DiffusionSSederr-5 and cuRobo-1 are 11x and 6x faster than MrNet with success threshold 8'. respectively, in terms of total time. Since both cuRobo and DiffusionSeeder use nvblox for ESDF reconstruction, we compare the planning time between cuRobo and DiffusionSeeder instead of the total time. On average, DiffusiinnSeeder-50 is 3x faster than cuRobo-1 1 with a 30% higher success rate. Compared to best-performing cuRobo-100 100, DiffussionSeeder-00 is 12x faster with a 10% higher success rate. We show the 75th and 98th quantile of the planning time of cuRobo- 100 and DiffusionSSeder-501 in both partially and fully observed environments (Appendix 7.8.6) in Figure 3, where cuRobo planning time has great discrepancy across the mean, 75th quantile, and 98th quantile. DiffusionSeeder-500 is 4x faster than cuRobo- 100 on the 75th quantile and 36x faster

---

Scenes
Success
Motion Time
cuRobo
DiffusionSeeder
cuRobo
DiffusionSeeder
Empty
5/5
5/5
34.6
40.4
Easy
(5/5,5/5)
(5/5,5/5)
39.5
43.3
Medium
(0/5,0/5)
(5/5,5/5)
41.5
Hard
(5/5,0/5)
(5/5,0/5)
40.8
44.1
Mean
57%
86%
38.0
42.3
Table Real experiment results for cuRobo and DiffusionSeeder. Each difficulty tier contains two scenes with 5 trials repeated for cach scene. We report number of successful trials for each seene in each tier. The method is successful if it doesn't 't collide with the environment across all trials in scene.
on 98th quantile in partially observed environments. While cuRobo shows high variance in planning time, DiffusionSeeder achieves relatively consistent performance among all scenes, indicating its advantage in generating scene-specific seeds.
5.2 Real Robot Evaluation
We conduct experiments on Franka Panda robot across 6 scenes, categorized into 3 difficulty tiers with 2 scenes from each tier and also an empty scene. Some of these scenes are shown in Figure 4. None the environment setups are part of our training dataset. In each scene, we select three poses that the robot needs to reach sequentially, repeating times. method is considered successful avoids collisions across all trials in scene. In addition to success, we also report the time the robot executing trajectories Motion Time. More details are in Appendix 7.9.
From Table 2, DiffusionSeeder failed once among the 7 scenes with an average success rate of 86% while cuRobo failed 3 times with an average success rate of 57%, demonstrating the sim2real transfer of DiffusionSeeder. DiffusionSeeder outperforms cuRobo on planning time with an average planning time of 26ms while cuRobo has a planning time of 65ms. The motion time of the planned trajectories of DiffusionSeeder is higher than that of cuRobo, similar to that in simulation exper- iments. As discussed in Appendix 7.8.5, hypothesize this may be attributed to the additional weight a on loss for non- non-linear trajectories, resulting DiffusionSSeder generate more non con-linear trajectories, which are less likely to collide but also have higher motion time. All failures for cuRobo was due to limited view of the obstacles from single camera. The one environment where Diffu- sionSeeder failed was because the tree obstacle did not fully fit in the view of the camera.
Conclusion and Limitation
We propose DiffusionSeeder, a diffusion- model for generating initial seed trajectories for mo- tion planning in novel scenes from just depth observations. Integrated with cuRobo, achieves up to 36x speedup on complex motion-planning tasks. DiffusionSeeder is trained on large-scale syn- thetic dataset but shows generalization to unknown real world scenes and observations. By utilizing an environment encoder and broad set of views of each scene during training, DiffusionSeeder generates collision-free seed trajectories under partial observation. In future, we hope to explore the capabilities of the environment encoder for single-vier view scene geometry understanding and investi- gate its applicability in other robotic domains, such as grasping and manipulation.
DiffusionSeeder has few limitations. It generates trajectories based on fixed external camera view, making it less effective when the goal pose occluded. We hope to extend DiffusionSeeder to incorporate multiple input views and more diverse camera poses in the future, to benefit from multi- camera or wrist-mounted camera setups. DiffusionSeeder is only trained on the Franka robot. We hope to extend DiffusionSeeder to different robots in the future. While we empirically observed that the nonlinearity introduced in Eq. 2 improves the performance, there a lack of thorough analysis on the effectiveness of Eq. 2 compared to a regular MSE loss. A comprehensive study on the loss function design for robotic motion planning can an interesting future direction.

---

References
[1] A. Millane, H. Oleynikova, E. Wirbel, R. Steiner, V. Ramasamy, D. Tingdahl, and R. Sieg- wart. nvblox: Gpu-accelerated incremental signed distance field mapping. arXiv preprint arXiv:2311.00626 2023.
[2] B. Sundaralingam, S. S. Hari, A. Fishman, C. Garrett, K. Van Wyk, V. Blukis, A. Millane, H. Oleynikova, Handa, Ramos, al. curobo: Parallelized collision-free minimum-jerk robot motion generation. arXiv preprint arXiv:2310.17274, 2023.
[3] Valale.. Rapidly-exploring random trees: A new tool for path planning. Research Report 98/1, 1998.
[4] L. E. Kavraki, P. Svestka, J.-C. Latombe, and M. H. Overmars. Probabilistic roadmaps for path planning in high- nigh-dimensional configuration spaces. IEEE Trans. Robotics and Automation, 12 (4):566-580, 1996.
[5] N. Ratliff, M. Zucker, J. A. Bagnell, and S. Srinivasa. Chomp: Gradient optimization tech- niques for efficient motion planning. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA), pages 489-4944 494, 2009.
[6] M. Kalakrishnan, Chitta, E. Theodorou, P. Pastor, and S. Schaal. Stomp: Stochastic tra- jectory optimization for motion planning. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA), pages 4569-4574, 2011.
[7] J. Schulman, Y. Duan, J. Ho, A. Lee, Awwal, H. Bradlow, J. Pan, S. Patil, K. Goldberg, and P. Abbeel. Motion planning with sequential convex optimization and convex collision checking. Int. Journal of Robotics Research (IJR)), 33(9):2251-1270) 2014.
[8] A. H. Qureshi, A. Simeonov, M. J. Bency, and M. C. Yip. Motion planning networks. In 2019 International Conference on Robotics and Automation (ICRA), pages 2118-2124, 2019. 10i: 109/1//.....129.9900
[9] B. Ichter and M. Pavone. Robot motion planning in learned latent spaces. IEEE Robotics and Automation Letters, 4(3):2407-2414, 2019.
[10] B. Ichter, J. Harrison, and M. Pavone. Learning sampling distributions for robot motion plan- ning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7087-7094. IEEE, 2018.
[11] A. Fishman, A. Murali, C. Eppner, B. Peele, B. Boots, and D. Fox. Motion policy networks. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
[12] A. H. Qureshi, Y. Miao, Simeonov, and M. C. Yip. Motion planning networks: Bridging the gap between learning-based based and classical motion planners. IEEE Trans. Robotics, 37(1): 48-66, 2021. doi: 10...1100/TTRO.2000000000
[13] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[14] C. Chi, S. Feng. Du, Z. Xu, E. Cousineau. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04117, 2023.
[15] M. Likhachev, J. Gordon, and Thrun. Ara*: Anytime a with provable bounds on sub- optimality. In NIPS, 2003.
[16] P. E. Hart, N. J. Nilsson, and B. Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE Trans. Syst. Sci. Cybern, 4:100-107, 1968.

---

[17] C. Zhang, J. Huh, and D. D. Lee. Learning implicit sampling distributions for motion planning. In 2018 TEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3654-3661. IEEE, 2018.
[18] C. Chamzas, Z. Kingston, C. Quintero-Pena, A. Shrivastava, and L. E. Kavraki. Learning sampling distributions using local 3d workspace decompositions for motion planning in high dimensions. In 2021 IEEE International Conference Robotics and Automation (ICRA). IEEE, May 2021. doi: 00011009//c44488562222211111
104.
[19] J. Wang, W. Chi, C. Li, C. Wang, and M. Q.-H. Meng. Neural rtth: Learning-based optimal path planning. 17(4):1748-1758 2020.
[20] M. Danielczuk, A. Mousavian, C. Eppner, and D. Fox. Object rearrangement using learned implicit collision functions. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA), pages 6010 6017, 2021.
[21] A. Murali, A. Mousavian, C. Eppner, A. Fishman, and D. Fox. Cabinet: Scaling neural col- lision detection for object rearrangement with procedural scene generation. In International Conference Robotics and Automation, 2023.
[22] G. Williams, A. Aldrich, and E. A. Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 2017.
[23] M. Bhardwaj, B. Sundaralingam, A. Mousavian, N. D. Ratliff, D. Fox, F. Ramos, and B. Boots. Storm: An integrated framework for fast joint-space model-predictive control for reactive ma- nipulation. In Conference on Robot Learning, pages 750-759. PMIR, 2022.
[24] Ichnowski, Y. Avigal, V. Satish, and K. Goldberg. Deep learning accelerate grasp- optimized motion planning. Science Robotics, 5(48))eabd7710, 2020. 12//scroooossss..tiiiiccss abd7710.
[25] M. Yoon, M. Kang, D. Park, and S.-E. Yoon. Learning- based initialization of trajectory optimization for path-following problems of redundant manipulators. In 2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 9686-9692, 2023. doi: 01009///CRR48811.222......
[26] M. Janner, Y. Du, B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0m) 099919 2022.
[27] Carvalho, T. Le, M. Baierl, Koert, and J. Peters. Motion planning diffusion: Learn- ing and planning of robot motions with diffusion models. In 2023 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1916-1923. IEEE, 2023.
[28] K. Saha, V. Mandadi, J. Reddy, A. Srikanth, A. Agarwal, B. Sen, A. Singh, and M. Kr- ishna. Edmp: Ensemble-of-com costooo--oooooeoeeeeeeooo -ost--s---dddd diffusion for motion planning, 2023. URL https: // //arxiv... org/ abs 2309. 11414.
[29] Huang, Z. Wang, P. Li, B. Jia, Liu, Y. Zhu, W. Liang, and S.-C. Zhu. Diffusion-based gen- eration, optimization, and planning 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16750-16761, 2023.
[30] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical im- age segmentation. In Medical image computing and computer-assisted InterventionMMCCAA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-244-4422 Springer, 2015.
[31] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
10

---

[32] D. Gammell, S. S. Srinivasa, and T. D. Barfoot. Batch informed trees (bit*): Sampling-basee optimal planning via the heuristically guided search of implicit random geometric graphs. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 3067-3074, 2015. doi:10.1 109/ICRA.221...136622..
[33] I. A. Şucan, M. Moll, and L. E. Kavraki. The Open Motion Planning Library. IEEE Robotics & Automation Magazine, 19(4):72-82, December 2012. 1o (09/MMAA.012.222555
[34] D. Coleman, I. Sucan, S. Chitta, and N. Correll. Reducing the barrier entry of complex robotic software: a moveit! case study. arXiv preprint arXiv:1404.3785, 2014.
[35] Michaux, A. Li, Q. Chen, Chen, B. Zhang, and R. Vasudevan. Safe planning for articulated robots using reachability- obstacle avoidance with spheres, 2024. URL https://arxiv, org/ ats//bb///00 08877

---

7 Appendix
7.1 Problem Generation
To increase data diversity, we sample additional pairs of start and end state poses near the original start and end poses. Since most scenes have obstacles near the end pose, we also sample problems near the end pose to encourage difficult trajectory generation. Specifically, for each scene, we obtain start pose in Cartesian space from the M*Net start state and sample N-1 new motion planning problems (start and end pose pairs) within a bounding box of 0.3m along the X- Y- and Z-axes around the original pose. We sample N new problems within bounding box of 0.3m near the end pose. We also sample N pairs of poses in free space, within cuboid of length 0.6m, increase our coverage over the joint configuration space of the robot. This results in 3N start and end pose pairs for each scene, We run inverse kinematics for each pose in the problem to filter out ones where either pose is in collision with the scene.
We pass each motion planning problem to cuRobo to generate smooth and collision-free trajectories with fixed trajectory length T 32. We define the trajectory length the number of waypoints and perform linear interpolation between the points. Problems for which cuRobo is unable to find a feasible solution are filtered out and discarded. With N = 5, we generated 15M problems with feasible and smooth solutions of size 32x7 on 3M unique scenes.
7.2 Environment Rendering
Each camera pose is represented by an azimuth a, an elevation e, and radius r. For each scene, we discretize orientations to a fixed grid of M -N. .N (where No are discretization resolution on azimuth and N₂ is discretization resolution
sphere centered scene mesh and pointing inward toward the center of the scene. Azimuths range is [-m/4, π /4]. elevations range is (TT /5, 2TT /5), and radius range is 007 x R, 1.1 x R] where R is the radius of scene mesh. We apply uniform random noise E [- f- /8, π. T/8 and E |-T/15, x/15) 15] to each discretized a, respectively to obtain diverse camera poses across different scenes.
elevation) camera poses
We render depth images from each camera pose and filter out camera poses where the problem end pose is not visible in the rendered depth images. The goal is visible if the projection of the goal in the camera view has larger depth than the goal itself-in other words, none of the observed pixels can occlude the goal in the camera frame. Using N. = 4, Ne = 3 and saving up to 4 images per scene (randomly sampled from the valid ones), we rendered 12M depth images of size 256 x 256 on 3M scenes.
7.3 Denoising Diffusion Probabilistic Models
A DDPM parameterized by & denoises given æı for K iterations to generate zo. where zo q(2))- J1.K are generated through diffusion process q(Fx|(x=-1), modeled as Markov chain, which gradually adds Gaussian noise Zg. The training objective of DDPM is therefore minimize the negative log likelihood E[- log (ogo))) which is upper bounded by [13]
Po (
Egi-logpo - Σ log g(zx|<R-1)
(3)
As the upper bound is calculated in expectation of the diffusion process 4(2)(24-1), we sample different time steps l E [o, K" during the training for estimating the expectation. The denoising process of DDPM defined as a Markov chain with learned Gaussian transitions such that
po(
Eo(xs, (o Eo (.),,))))
As shown in [13], the parameterization of the denoising process be modified to train noise prediction model to predict the noise from Zg.
12

---

Figure 5: Example train (left) and test (right) problems three scene types: cubby, tabletop, and dresser (from left to right in the left image). The robot in the start configuration and the target end pose is marked in green.
7.4 Model Training
During training, for each problem, we sample one depth image from the pre-rendered depth images. We also reverse the start and end pose, and the order of the ground truth trajectories half of the time generate problems going from the end to the start, for further data augmentation. The ground truth trajectories and the input robot start configuration the joint position by the corresponding joint mechanical limit, to improve the training stability. We then denormalize the output of the diffusion model by multiplying with the joint mechanical limit.
normalized to be from 0 to 1 by dividing
7.5 Model Inference
As the diffusion and denoising process of DDPM are both Markovian, DDPMs generate by denoising K steps from IK, resulting in longer inference time. Denoising Diffusion Implicit Models (DDIM) [31] are an efficient class of iterative implicit probabilistic models. They have the same training objective as DDPMs but model non-Markovian diffusion processes. DDIMs thus allow the diffusion model trained with DDPM to generate æx directly from T without the generation of the intermediate steps, speeding up the inference time by greatly reducing the number of denoising iterations. During inference time, randomly sample noise vector of the same shape as the trajectory, and use DDIM generate a denoised trajectory of shape 32x 7, We convert the trained model to BFLOATI6 for accelerated inference and implement the DDIM step as fused CUDA kernel leveraging JIT scripting in Pytorch 2.0. We encode the full inference step in a single CUDA Graph to further reduce python overheads. These techniques give us a 6ms inference time on an Nvidia GeForce RTX 4090, which includes the inference of the vision encoder, followed by steps of denoising.
7.6 Optimization Problem
use the trajectory optimization problem formulated cuRobo [2], which we write briefly below,
min Cgent (x. F (gr)) + Csmooahh (7)
+ Cseir(T) + (woela(T)
s.t. joint limits
where Cgoat is the cost for reaching the goal pose the final timestep T, Csmooth is term to encourage smooth minimum-jerk trajectories, Cseir and Cwardd are self and world collision avoidance cost terms.
7.7 Metrics
We consider the following quantitative metrics:
13

---

Figure 6: Example scenes and observations for narrow passage problems. goal poses are marked in green.
The robot is at start state and the
Metrics
Success Rate
Jerk (rad/s) s²)
Motion Time(s)
Err(mm)
Q Err(")
DiffusionSeeder cuRobo
52.4%
65.1
1.45
0.08
0.73
52.4%
100.33
1.17
0.19
0.91
Table 3: Experiments on 500 narrow passage problems for Seeder and cuRobo with partial observa- tions. As DiffusionSecder is never trained on such scenes, the performance of both methods are similar.
Success Rate: the percentage of collision-fres trajectories with the end pose translation error and quaternion error lower than & and 6, respectively.
Plan Time: the time the planner generate the trajectories given the observation.
Jerk: the maximum jerk of the flattened trajectories over all time steps and robot joints.
Translation Error: the mean squared error between the achieved end pose position and the target pose position.
Quaternion Error: the cosine distance between the achieved end pose quaternion and desired end pose quaternion.
Motion Time: overall execution time of the trajectories accounting for a robot's mechani- cal limits, such maximum velocity, acceleration, and jerk.
7.8 Simulation Experiments
We represent scenes with single depth image. Both cuRobo and DiffusionSeeder use nvblox with the given depth image for collision checking. As notice the performance of nvblox is sensitive to the image size, render depth images of size 640 x 640 at the test time for nvblox EDSF reconstruction and downsample them to 256 256 as the input to the diffusion model.
7.8.1 Evaluation on Narrow Passage Problems
To evaluate the generalization ability of DiffusionSeeder, evaluate DiffusionSeeder narrow passage problems. We generate 500 problems on 50 scenes consisting of 10 cuboid obstacles from the sparrow dataset [35]. Results are shown in Table 3. Both DiffusionSeeder and cuRobo achieve a success rate of 52.4% when given partial observation of depth images. Examples the problems and the depth observations are shown in Figure 6. As DiffusionSeeder never trained on such scenes, the performance of DiffusionSeeder similar to cuRobo. In the future work, we expect to include more diverse scene types in the training of DiffusionSeeder.
Metrics
Success Rate
Jerk (rad/s²)
Motion Time(s)
Err(ma))
Q Emr())
DifeerionSeeder
80.6%
77.0
1.26
0.57
1.73
DillusionSSeder with Guidance
81.1 %
76.7
1.27
0.60
1.77
Table 4: Experiments 1791 test problems for DiffusionSeeder with and without guidance with partial obser- vations. Cost gradient guidance improves the success rate of DiffusiinSeeder marginally.
14

---

7.8.2 Case Study for Scene Occlusion
For robotic motion planning, the same observations may bring different levels of difficulties for different motion planning problems, depending the information needed for solving the problem. Therefore, to systematically evaluate the occlusion, we conduct case study 50 sampled camera views. We sample 56 problems for this scene. For each observation, we run both DiffusionSeeder and cuRobo on all 56 problems and calculate the average success rate. The success rate of each method for different camera views are summarized in Figure 7. On average of all 50 camera views, DiffusionSeeder achieved success rate of 65.9% while cuRobo achieved a success rate of 60.2% From the Figure, the success rate of DiffusionSeeder is comparable with cuRobo when the scene is less occluded (green region), while DiffusionSeeder performs better than cuRobo when the scene is more occluded (yellow region). This indicates DiffusionSeeder has implicit scene completion ability and can plan better when the scene is heavily occluded.
cubby scene with
100
Success Rate (%)
75
50
25
Diffusion Seeder
cuRobo
0
Camera View Index
Figure 7: Success rate over 56 problems on cubby scene for DiffusionSeeder (red) and cuRoho (blue) for 50 different camera views. Depth observation examples are shown on the top. In general. the observations in the yellow region occluded while those in the green region less occluded.
7.8.3 Cost Guided Diffusion Sampling
Similar to [27, 28], we evaluate the benefits of guiding diffusion sampling with the cost gradient of the current scene. Specifically, we have
Th-1 ((, + ad,C, E.)
where is the diffusion denoising time step, A4 the reverse diffusion, Σg covariance schedule, A,C is the gradient of the cost with respect the current sampled trajectory pa po( and is weight scalar. As in [27], we implement the cost gradient guidance for t < k and take gradient steps for each time step t. We also notice excluding the guidance for the last denoising step improves the performance.
We evaluate DiffusionSeeder with and without guidance on the 1791 test problems with partial ob- servations. We set k 60, 20 and a 1, which achieves the best performance through parameter sweeping experiment. Results are summarized in Table 4. Notice the results of Diffu- sionSeeder is different from that reported in Table as more recent version of cuRobo (v0.7.4) is
15

---

used. From the results, the gradient guidance improves the performance marginally. We suspect that the improvement is marginal because DiffusionSeeder is conditioned on the scene and can already output seed trajectories at the low cost region of the scene. We leave how to better incorporate the gradient guidance the diffusion denoising process to future work.
7.8.4 Optimization Steps Analysis
From Table 1, DiffusionSSeder-445 has the highest success rate of 86.2% and DiffusionSSeder-25 has the lowest planning time of 1 15ms, with a success rate of 85.1% %. The trajectory quality increases with Niess indicated by the decreasing jerk, motion time, pose translation, and quaternion error, at the cost of increasing planning time from 15ms to 45mss. We observe that the success rate doesn't tt vary significantly with Nierr- We hypothesize that this is because the diffusion model is trained on the trajectories optimized for motion planning cost on the ground truth mesh and additionally sees multiple different views of the scene during training. It implicitly minimizes the trajectory opti- mization cost and can do implicit scene completion from partial observations. Therefore, the seed trajectories are closer to the optimal collision-fin -fre trajectory while avoiding collision in unobserv- able areas and require fewer optimization steps to generate collision-free trajectories. Contrarily, cuRobo optimizes on trajectories generated by a graph-baser planner or linear interpolation that are less optimal, requiring more optimization iterations to minimize the motion optimization cost.
7.8.5 Trajectory Quality Comparisons
Among all methods, BiTStar has the lowest jerk but also the lowest success rate. MrNet has the highest motion time and target pose tracking error. The maximum jerk of DiffusionSeeder gen- erated trajectories is slightly higher than cuRobo but still far below the robot's mechanical limit. DiffusionSeeder has slightly higher motion time, likely coming from the non-linear trajectories that cuRobo fails to solve. The additional weight on loss for non-linear trajectories (see Section 4.3) may also contribute to this high motion time DiffusionSeeder. Both DiffusionSeeder and cuRobo have sub-millimeter target pose translation tracking error and low quaternion tracking error.
7.8.6 Full vs Partial Observability
Metric
BiTStar
McNNt
cuRabo v0l622
DiftusionSeader
Condition
5
5
.arp
Narp
Nap
Nitura
50
Necrz
Niterx
Plan
031
0.45
0154
00065
0.066
(U)14
0:030
0,027
48.4%
99.77%
96.7%
Jark (rnd/x²)
59.7
106.0
113.44
97.4
80.1
71.5
Mation (s)
2.57
4.78
1:25
1.20
1.31
Translation Ear (mm)
3.98
3.92
7.16
3:28
0.1
0.1
0.1
0.03
0.07
0.05
0.03
Quaternion Eat(") )
13.07
0.92
3.36
2:23
0.55
0.54
0.53
1.99
1.36
0.29
0.17
Table 5: Evaluation with ground truth mesh for BiTStar, MrNet, cuRobo and DiffusionSeeder. Mean values of
each metric over the 1791 test problems are reported. Same as experiments with partial observations, we report results of BiTStar and MrNet with different success threshold and cuRobo with different number of attempts and DiffusionSeeder with different number of optimization iterations.
In this set of experiments, we pass the ground truth mesh for optimizer collision checking for BiT- Star, cuRobo, and DiffusionSeederr. For MrNet, we pass point clouds sampled from the ground truth mesh as in M*Net. We calculate each metric for successful trajectories, with the average value summarized in Table 5, From Table BiTStar has a success rate of 61% under success threshold 5. providing benchmark of the difficulty of the problems. cuRobo- 100 has the highest success rate of 99.7%, while DiffusionSeeder-500 has success rate of 97%. It's worth noting that the diffusion model of DiffusionSeeder still takes partial observations of a depth image while cuRobo takes in full observations, which explains the slightly lower success rate of DiffusionSSeder compared to cuRobo and the consistent success rate across different Nuans The trajectory quality with Nuse shows simi- lar trends to those partially observed environments. In fully observed environments, cuRobo-100 still significantly slower than DiffusionSeeder as shown in Figure 3b, where DiffusionnSeder-500 is 6x faster average, 4x faster 75th quantile and 22x faster the 98th quantile. As afore- mentioned, cuRobo calls graph based planner when the linear interpolation trajectory fails for
16

---

complicated problems, resulting in the significant slow down on obstacle-dense scenes. In contrast, DiffusionSeeder has a consistent planning time over all scene types.
7.8.7 Impact of Nirajs
We study the performance of DiffusionSeeder with Nuajs in partially observed environments. We set Nuers = 200. Results are shown in Table 6. The success rate of DiffusionSeeder increases monotonically upto Niajs 12, suggesting trajs 12 is able to represent the potential modalities sufficiently. The success rate of Nuajs 12 is 1.99x higher than that of Nirjs 1, highlighting the benefits of using the diffusion model generate multiple diverse seeds, compared to motion initial trajectory through multi-step rollout. We notice that the
planning policies that generate success rate is not increasing monotonically after Nuajs = 12. We hypothesize this might be because as Mijj increases, the ratio of linear trajectories also increases, resulting the optimizer to optimize a more linear trajectory, which is implied by the lower motion time for higher Nuajs
Neyss
12
Plan Time (8)
0.023
0.039
0.030
0.027
0.027
0.026
0.04
Success Rate
42.8%
47.5%
70.7%
77.7%
85.1%
82.7%
84.3%
Jerk (rad/a") s²)
79.5
66.4
74.6
84.7
93.5
84,2
85.5
Motion Time (s)
1.32
1.33
1.30
1.30
1.24
Translation (mm)
0.35
0.27
0.36
0.51
0.50
0.41
0.45
Quaternion Er (*)
0.68
0.59
0.78
0.90
1.03
0.86
1.71
Table 6: Evaluation with partial observations of a 640x640 depth image for DiffusionSeedor with diffcrent
Ntroja- Mean values of each metric over the 1791 test problems are reported. The performance increases
monotonically until Neroja = 12.
7.8.8 Impact of Number of Views
DiffusionSeeder can take in multiple depth images by design. Though we only train the model on one depth image, we can provide multiple depth images at the inference time. We set Niters 200 and Nunjs = 12 for DiffusionSeeder and provide 1, 2, and 5 depth images. As shown in Table 7, in- creasing number of camera views does not improve the performance, likely because DiffusionSeeder only trained on one depth image.
Num Views
2
Plan Time (s)
0.027
0.025
0.024
Success Rate
85.1%
84.5%
83.8%
Jerk (rad/s"
93.5
93.1
88.2
Motion Time (s)
1.30
1.23
1.23
Translation Err (mm)
0.50
0.81
0.69
Quaternion Err (9)
1.03
1.75
1.62
Table Evaluation of DiffusionSeeder with different number of 640x640 depth images. Mcan values of cach metric over the 1791 test problems are reported.
7.8.9 Performance of DiffusionSeeder without cuRobo
We evaluate the performance of DiffusionSeeder without integration of cuRobo (i.e., trajectory optimization). We sample Nuajs from the diffusion model and choose the trajectory that has the lowest translation eтооr as the final solution. We do collision checking with the ground truth mesh and check if the translation error and quaternion error is below the success threshold to calculate the success rate.
Results are summarized in Table As Nmjj increases, the success rate increases monotonically. The collision free rate, which is consistent with respect the Nugjs is the upper limit of the success rate of the diffusion model as collision-fiee trajectories can have large end pose errors. The success rate
17

---

Nirajs
S.R.
CFR
Jerk (rad/s²)
M.T. (8)
Err (mm)
Err ())
13.2%
38.1%
291.5
1.36
3.37
2.55
12
15.4%
37.9%
289.9
1.36
3.28
2.60
16
17.2%
37.8%
286.2
1.37
3.22
2.64
32
21.1%
38.0%
280.6
1.40
3.13
2.62
64
22.4%
37.7%
282.5
1.39
2.86
2.72
128
26.2%
38.0%
273.0
1,42
2.70
2.66
256
27.6%
38.0%
269.1
1,43
2.47
2.76
512
29.3%
38.0%
268.1
1,43
2.29
2.79
1024
31.5%
37.9%
268.4
1.43
2.13
2.88
Table 8: Performance of DiffusionSeeder without cuRoho. We report Success Rate (S.R.), the collision-
free Rate (CFR), the maximum Jerk, the Motion Time (M.T.), the Translation Error (T Err) and the Quaternion Error (Q Err). The collision-fiet free rate is defined as the number of the collision-fit ffeee trajectories among all Nuajs sampled trajectories.
of the diffusion model is much higher than MrNet (Table 1), indicating that the diffusion model is able to generate environment dependent seeds. However, the success rate is much lower than that of DiffusionSeeder integrated with cuRobo, highlighting the benefits optimizing the diffusion seeds with explicit collision checking using nvblox and other cost terms. In addition, diffusion generated trajectories tend to have high maximum jerk around 270 rad/s which significantly reduced by cuRobo optimization around 106 rad/s².
7.9 Real Robot Evaluation
We use a Realsense D435 depth camera mounted opposite to the robot to obtain depth images. The robot is segmented and removed from the depth images and then sent to DiffusionSeeder and nvblox. None of the environment setups are part of our training dataset, in addition the chosen camera view causes severe occlusions in the environment when obstacles are present as seen in Figure 1. Hence we used Niter 100 DiffusionSeeder as using did not get us high-quality solutions. For cuRobo, we use the default number of iterations (475) provided in their code base and have Nap 10. For both methods, run the Franka Panda with joint impedance controller so that collisions in the world do not damage the robot. In addition, we set the maximum allowed velocity to 65% of the robot's limits across the methods to avoid any catastrophic damage due to occlusions in the world.
18


Reviews:
[{"review_id": "sn2dDUIFeJ", "reviewer": "Reviewer_kGxy", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes diffusion model-based trajectory generation, which can be used to improve planning efficiency and the success rate of trajectory optimization in complex environments. The method additionally addresses handling partial observability by considering environment conditioning with a depth image obtained by a fixed camera. The proposed method shows promising results compared to various baselines in both planning time and success rate.", "summary_of_recommendation": "The paper's contribution appears significant, and the presentation and technical details are clear and sufficient.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Please address the two comments in the weaknesses above.\n- In Figure 1’s caption, “nvblox [1] online from the depth images …” What does \"online\" mean here? I thought the overall pipeline computes a trajectory offline, so this part was confusing. \n- Figure 2 does not include the “time step” latent vector.\n- In Section 5.1: 575 dim -> 576 dim\n- At the end of Section 5.1: “831-dimensional conditional vector for each layer.” Which layer does this refer to?\n- In Figure 4, mark the goals. Otherwise, it’s hard to judge whether the problem is easy or hard.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oFwEh0to1S", "reviewer": "Reviewer_QoEm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper propose to use diffusion method to generate conditional initial trajectories for trajectory optimization.", "summary_of_recommendation": "This work propose to use ViT conditioned difffusion to generate initial planning paths and demonstrates its effectiveness on a large dataset. The method can have a good impact for further large scaled robot learning.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. I noticed the experiments in the video are either the end effector going up and going down, or the end effector going into the cabinet. Could this method work for more challenging narrow passage problems? For example, multiple links go into the cabinet.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sT51f2b0kT", "reviewer": "Reviewer_oM7J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "DiffusionSeeder uses a conditional Denoising Diffusion Probabilistic Model (DDPM) to generate diverse, high-quality seed trajectories from initial depth images of the environment. These seed trajectories are then optimized using cuRobo, a GPU-accelerated motion optimizer. The key claim is that DiffusionSeeder accelerates motion planning and increases success rates in complex environments with image observations by providing high-quality seed trajectories for trajectory optimization, trained on big datasets. The evaluation involves both simulated and real-world experiments using the Franka robot. In simulations, DiffusionSeeder is tested on 3M scenes from the MπNet dataset, comparing performance with BiTStar, MπNet, and cuRobo.", "summary_of_recommendation": "This works has clear motivations and clear engineering description. However, I am not totally convinced on the experiment statistics. I will increase my score when additional experiment results added after the rebuttal.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Why does DiffusionSeed have high jerks compared to a sampling-based approach such as BitStar?\n2. Please compare DiffusionSeeder to [1], with the focus on comparing two paradigms of coupling vs. decoupling diffusion prior update and likelihood (i.e., planner) gradient update, to see if the best possible method configuration of diffusion prior for motion planning.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes the use of the diffusion method to generate conditional initial trajectories for trajectory optimization. Overall, the paper is well-written and easy to follow. However, it is necessary to demonstrate the proposed work in more challenging narrow passages to validate its capability. Furthermore, a comparison against relevant baselines such as DiffusionSeede is necessary to validate the contribution of the proposed approach over prior work. Additionally, the results should be clearly explained, including why the proposed method has more jerks in the trajectories, and the design of the objective function also needs to be motivated.

Update: The revised version has addressed the reviewer's comments to a good extent.
Decision: Accept