Motion Policy Networks
Authors: Adam Fishman
Adithyavairavan Murali
Clemens Eppner
Bryan Peele
Byron Boots
Dieter Fox
Venue: CoRL 2022 Poster (2022)
Keywords: Motion Control
Imitation Learning
End-to-End Learning

Abstract: Collision-free motion generation in unknown environments is a core building block for robot manipulation. 
Generating such motions is challenging due to multiple objectives; not only should the solutions be optimal, the motion generator itself must be fast enough for real-time performance and reliable enough for practical deployment.
A wide variety of methods have been proposed ranging from local controllers to global planners, often being combined to offset their shortcomings. We present an end-to-end neural model called Motion Policy Networks (M$\pi$Nets) to generate collision-free, smooth motion from just a single depth camera observation. M$\pi$Nets are trained on over 3 million motion planning problems in more than 500,000 environments. Our experiments show that M$\pi$Nets are significantly faster than global planners while exhibiting the reactivity needed to deal with dynamic scenes. They are 46% better than prior neural planners and more robust than local control policies. Despite being only trained in simulation, M$\pi$Nets transfer well to the real robot with noisy partial point clouds. Videos and code are available at https://mpinets.github.io

Motion Policy Networks
Adam Fishman 1.a
Adibyyavairavn Murali 2 Clemens Eppner* 2
Bryan Peele Byron Boots 1,2 Dieter Fox
University of Washington 2NVIDIA (ffishman, bboots, fox 0cs washington. edu (admurali, ceppner, bpeele Anvidia.. com
https://mpinets.giibae
Collision-free motion generation unknown environments core building block for robot manipulation. Generating such motions is challenging due to multiple objectives; not only should the solutions be optimal, the motion generator itself must be fast enough for real-time performance and reliable enough for practical deployment. A wide variety of methods have been proposed ranging from local controllers global planners, often being combined to offset their shortcomings. We present an end-to-end neural model called Motion Policy Net- works (MrNets) to generate collision-free, smooth motion from just a single depth camera observation. MrNets are trained on over 3 million motion planning prob- lems in more than 500,000 environments. Our experiments show that MrNets are significantly faster than global planners while exhibiting the reactivity needed to deal with dynamic scenes. They are 46% better than prior neural planners and more robust than local control policies. Despite being only trained in simulation, MrNett transfer well to the real robot with noisy partial point clouds. Videos and code are available at https: //mpinets..../////m github. io.
Abstract:
Keywords: Motion Control, Imitation Learning, End-to-end Learning
Figure 1: MrNets are trained a large dataset of synthetic demonstrations (leff) and can solve complex motion planning problems using raw point cloud observations (right).
Introduction
Generating fast and legible motions for robotic manipulator in unknown environments is still an open problem. Decades research have established many well-studied algorithms, but there are two practical issues that prevent motion planning methods from being widely adopted in industrial applications and home environments that require real-time control. First, it challenging for any single approach to satisfy multiple planning considerations: speed, completeness, optimality, ease- of-use, legibility (from the perspective human operator), determinism, and smoothness. Second, existing approaches enforce strong assumptions about the visual obstacle representations succh as accurate collision checking in configuration space [1] or availability of gradient [2, 3, 4]-and hence require expensive intermediate processing of sensor readings to operate in novel scenes.
Work done while the author was an intern at NVIDIA.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Some planners, such as RRT [5], are useful to quickly find a feasible path. Other methods are useful to find an optimal path, either deterministically on discrete graphs [6, 7] or asymptotically in continuous spaces [8, 9]. Some planning techniques use Neural Networks [10, 11] to improve methods [2, 3] are
sampling efficiency, thus speeding up the planning process. Optimization- useful to encode other objectives, such as smoothness, at the expense of guarantees. Going further, recent motion generation frameworks [4, 12] eschew long-term reasoning in exchange for quick local decisions under the assumption that they will lead to globally acceptable paths.
Each of these methods requires known enviromment model and perfect state estimation. In practice, one would have to create scene representation, which could be a static or dynamic mesh, an occupancy grid [11. 13], signed distance field, etc. Reconstruction systems such as SLAM and KinectFusion [14] have a large system start-up time, require a moving camera to aggregate many viewpoints, and ultimately require costly updates the presence of dynamic objects. Recent implicit deep learning methods like DeepSDF [15] and NeRF [16] are slow or do not yet generalize to novel scenes. Methods such as SceneCollisionNet [17] provide fast collision checks but have not yet been shown to generalize to challenging environments beyond a tabletop. Other RL-based methods learn a latent representation from observations but have only been applied to simple 2D [18, 19] or 3D [20] environments in simulation.
We present Motion Policy Networks (MmNets), a novel method for learning an end-to-end policy for motion planning. Our approach circumvents the challenges of traditional motion planning and is flexible enough be applied in unknown environments. Our contributions are as follows:
We present a large-scale effort in neural motion planning for manipulation. Specifically, we learn from over 3 million motion planning problems across over 500,000 instances of three types of environments, nearly 300x larger than prior work [10].
We train reactive, end-to-end neural policy that operates point clouds of the environment and moves to task space targets while avoiding obstacles. Our policy is significantly faster than other baseline configuration space planners and succeeds more than local task space controllers.
On our challenging dataset benchmarks, we show that MaNets is nearly 46% more successful at finding collision-free paths than prior work [10] without even needing the full scene collision model.
Finally, we demonstrate sim2real transfer to real robot partial point cloud observations.
2
Related Work
Global Planning: Robotic motion planning typically splits into three camps: search, sampling, and optimization-basee planning. Each algorithmic family has benefits and drawbacks. Search-based planning algorithms, such as A [6. 7, 21] are complete, fast, and optimal in discrete domains. Sampling-based methods, c.g. [5. 8]. operate in continuous domains, but are only probabilistically complete, i.e. find a solution with probability 1. Some such methods are also asymptotically optimal [8, 22, 9]. but within practical time limitations produce sub-optimal-and sometimes erratic-paths. Motion Optimization [2, 3, 23] can produce smooth paths to a goal but is prone to local minima. Without careful system design, often on per-task basis, Motion Optimization can fail to find the optimal solution or sometimes any solution at all. See Appendix A for a deeper discussion on the varying advantages and disadvantages of each algorithmic family.
Local Control: In contrast to global planners, local controllers have long been applied to create collision-free motions [24, 25, 4, 12]. While they prioritize speed and smoothness, they are highly local and may fail to find a valid path in complex environments. We demonstrate in our experiments that MrNets are more effective at producing convergent motions in these types of environments, including in dynamic and in partially observed settings.
Imitation Learning: Imitation Learning [26] can train a policy from expert demonstrations with limited knowledge of the expert's internal model. For motion planning problems, we can apply imitation learning and leverage a traditional planner as the expert demonstrator-wint
perfect model of the scene during training-and learn a policy that forgoes the need for an explicit scene model at test time. Popular imitation learning methods include Inverse Reinforcement Learning [27, 28, 29] and Behavior Cloning [30, 31]. See Appendix A for a more detailed discussion of these methods and their trade-offs. Our method seeks to overcome the common challenges of Behavior

---

Cloning by specifically designing learnable expert, increasing the scale and variation of the data, and using sufficiently expressive policy model.
Many deep planning methods [11, 32, 33, 34] seek to learn efficient sam- plers to speed up traditional planners. Motion Planning Networks (MPNets) [10] learn directly plan through imitation of a standard sampling-base baaiin RRT planner [8] and is used in conjunction with traditional planner for stronger guarantees. While these works greatly improve the speed of the planning search, they have the same requirements as a standard planning system: targets in con- figuration space and an explicit collision checker to connect the path. Our work operates based on task-space targets and perceptual observations from a depth sensor without explicit state estimation.
Neural Motion Planning:
Novel architectures have been proposed, such differentiable planning modules in Value Iteration Networks [19), transformers by Chaplot et al. [35] and goal-conditioned RI policies [36]. These methods are challenging to generalize unknown environments have only been shown in simple 2D [18] or 3D settings [20]. In contrast, we illustrate our approach the challenging domain of controlling a degrees of freedom (DOF) manipulator in unknown, dynamic environments.
Learning from Motion Planning
3.1 Problem Formulation
McNets expect two inputs, robot configuration qi and a segmented, calibrated point cloud Before passing qi through the network, we normalize each element be within [-1, 1] according to the limits for the corresponding joint. We call this q."" The point cloud is always assumed to be calibrated in the robot's base frame, and it encodes three segmentation classes: the robot's current geometry, the scene geometry, and the target pose. Targets are inserted into the point cloud via points sampled from the mesh floating end effector placed at the target pose.
The network produces a displacement within normalized configuration space a!!:!. To get the next predicted state Φ++1. take q! + o!!'! clamp between [-1, 1]. and unnormalize. During training, we use @++1 to compute the loss, and when executing, we use @++1 as the next position target for the robot's low-level controller.
3.2 Model Architecture
Configuration Enooder
Pointelond Encoder
Latent
Decoder
9
Robos
Space
Obstacles
Farget
Figure 2: MrNets encodes state as normalized robot configuration and segmented point cloud with three classes for the robot, the obstacles, and the target. The policy outputs a displacement in normalized joint space, which can then be applied to the input before unnormalizing to get q1+1-
The network consists of two separate encoders, one for the point cloud and one for the robot's current configuration, as well as a decoder, totaling 19M parameters. Our neural policy architecture visualized in Fig. We use PointNet++ [37] for our point cloud encoder. PointNet++ learns hierarchical point cloud representation and can encode a point cloud's 3D geometry, even with high variation in sampling density. PointNet++ architectures have been shown to be effective for a variety of point cloud processing tasks, such as segmentation [37]. collision checking [17], and robotic grasping [38, 39]. Additionally, PointNet++ includes PointNet as a subcomponent. PointNet is effective at processing partially observed point clouds, even when trained exclusively with fully- observed scenes [40]. The robot configuration encoder and the displacement decoder are both fully connected multilayer perceptrons. We discuss the architecture in detail in Appendix D.

---

3.3 Loss Function
The network trained with a compound loss function with two constituent parts: behavior cloning loss to enforce accurate predictions and a collision loss to safeguard against catastrophic behavior.
Geometric Loss for Behavior Cloning To encourage alignment between the prediction and the expert, we compute a geometric loss across a set of 1,024 fixed points along the surface of the robot.
f/41 = 0' Âq:) dlt1 o" ('(+)))
LBC(Aqq) E1 2/tilla + |1 2/411l1. where
'' represents forward kinematics mapping from the joint angles of the robot to point defined on the robot's surface. The loss is computed as the sum of the L1 and 1.2 distances between cor- responding points the expert and the prediction after applying the predicted displacement. By using both L1 and L2, we are able to penalize both large and small deviations.
We use a geometric, task-space loss because our goal is to ensure task-space consistency of our policy. Configuration space loss appears prior work [10], but does not capture the accumulated error of the kinematic chain as effectively (see Appendix K).
Collision Loss In order to avoid collisions-a catastrophic failure we apply an additional hinge- loss inspired by motion optimization [41].
where hy(41+1) ( (D2 10.
if D2 0 if D2(
Lrcolissonn = ΣΣ h
(2)
The synthetic environments are fully-observable during training, giving us access to the signed- distance functions (SDF), {D,(-)}, of the obstacles in each scene. For given closed surface, its SDF maps point in Euclidean space to the minimum distance from the point to the surface. If the point is inside the surface, the function returns a negative value.
Procedural Data Generation
4.1 Large-scale Motion Planning Problems
Each planning problem defined by three components: the scene geometry, the start configuration, and the goal pose. Our dataset consists of randomly generated problems across all three components, totaling 3.27 million problems over 575, 000 environments. We have three classes of problems of increasing difficulty: cluttered tabletop with randomly placed objects, cubbies, and dressers. Rep- resentative examples of these environments are shown in Fig. 1. Once we build these environments, generate set of potential end-effector targets and corresponding inverse kinematies solutions. We then randomly choose pairs of these configurations and verify if a plan exists between them using our expert pipeline, detailed further in Sec. 4.2 and in Appendix E.
4.2 Expert Pipeline
Our expert pipeline designed to produce high-quality demonstrations we want to mimic, i.e. tra- jectories with smooth, consistent motion and short path lengths. Here, consistency is meant to describe quality and repeatability an expert planner-see see Appendix C for further discussion. We considered two candidates for the expert the Global Planner which is typical state-of-the-an configuration space planning pipeline [42] and a Hybrid Planner that we engineered specifically to generate consistent motion in task space. For both planners, we reject any trajectories that produce collisions, exceed the joint limits, exhibit erratic behavior (i.e. high jerk), or that have divergent motion (i.e. final task space pose is more than 5cm from the target).
Global Planner consists of off-the-sheif components of standard motion planning pipeline-inverse kinematics (IK) [43], configuration-space AIT [42], and spline-based, collision-aware trajectory smoothing [44]. For solvable problem, as the planning time approaches infinity, IK will find valid solution and A will produce an optimal collision-free path, both with probability 1. Likewise,

---

with continuous collision checking, the smoother will produce smooth, collision-fis cellsson-ffree path. In practice, our dataset size goal we generated 6.54M trajectories across over 773K environments dictated our computation budget and we tuned the algorithms according to this limit. We attempted IK at most 1, 1,000 times, utilized an AIT time out of 20s, and employed discrete collision checking when smoothing. Most commonly, the pipeline failed when AIT* timed out or when, close to obstacles, the smoother's discrete checker missed a collision, thereby creating invalid trajectories.
Hybrid Planner is designed to produce consistent motion in task space. The planner consists of task-space AIT [42] and Geometric Fabrics [4]. AIT* produces an efficient end effector path and Geometric Fabries produce geometrically consistent motion. The end effector path acts as dense sequence of waypoints for sequence of Geometric Fabrics, but as the robot moves through the waypoints, the speed can vary. To promote smooth configuration space velocity over the final trajectory, we fit a spline to the path and retime to have steady velocity. As we discuss in Sec. 5.1, Geometric Fabrics often fail to converge to a target, so we redefine the planning problem to have the same target as the final position of the trajectory produced by the expert. Inspired by [45], we call this technique Hindsight Goal Revision (HGR) and demonstrate its importance in Sec. 5.4. Using the Hybrid Planner, we generated 3.27 million trajectories across 576,532 environments.
5 Experimental Evaluation
We evaluate our method with problems generated from the same distribution as the training set. See Appendix E for more detail on the procedural generation and random distribution. Within the test set, each problem has a unique, randomly generated environment, as well as unique target and starting configuration. None of the test environments, starting configurations, nor goals were seen by the network during training. Our evaluations were performed on three test sets: a set of problems solvable by the Global Planner, problems solvable by the Hybrid Planner, and problems solvable by both. Each test set has 1,800 problems, with 600 in each of the three types of environments.
roll it out until it matches one of two termination conditions: 1) the Euclidean distance to the target is within lcm or 2) the trajectory has been executed for s (based consultations with the authors of [4] and [12]). We consider the following metrics (see Appendix H for details):
Quantitative Metrics:
To understand the performance of policy,
Success Rate A trajectory is considered a success if its final position and orientation target emrors are below em and 15t respectively and there are physical violations.
Time We measure the wall time for each successful trajectory. We also measure Cold Start (CS) Time, the average time to react to a new planning problem.
Rollout Target Error The L2 position and orientation error (taken from [46]) between the target and final end-effector pose in the trajectory.
Collision Rate The rate of fatal collisions, both self and scene collisions
Smoothness We use Spectral Are Length (SPARC) [47] and consider a path to be smooth its SPARC values in joint and end-effector space are below 11.6.
5.1 Comparison to Methods With Complete State
Most methods to generate motion in the literature assume access to complete state information in order to perform collision checks. In each of the following experiments, we provide each baseline method with an oracle collision checker. When running MrNets, we use a point cloud sampled uniformly from the surface of the entire scene. Results are shown in Table 1.
Global Configuration Space Planner The Global Planner is unmatched in its ability to reach target, but this comes the cost of average computation time (16.46s) compared to Mr.Net (0.33s). With global planner, there is no option to partially solve a problem, meaning the Cold Start Time equal to the planning time. In a real system, optimizers [2. 3, 48] could be used to quickly replan once an initial plan has been discovered. As discussed in Sec. 4.2, the Global Planner is theoretically complete but fails in practice on some of the Hybrid Planner-solvable problems due to system timeouts and discrete collision checking during smoothing.
5

---

Success Rate (%)
Soln. Time (s)
CS Time (s)
Global
Hybrid
Both
Smooth (%)
Global Planner [42]
16.46 0.90
16.46 + 0.90
100
78.44
100
51.00
Hybrid Planner
7.37 + 2.23
7.37 + 2.23
50.22
100
100
99.26
G. Fabrics [4]
0.15 0.09
2.4e-4 + 3e-5
38.44
59.33
60.06
85.39
STORM [12]
4.03 + 1.89
13.4e-3 + 2.2e-3
50.22
74.50
76.00
62.26
MPNets [10]
Hybrid Expert
4.95 + 23.51
4.95 + 23.51
41.33
65.28
67.67
99.97
Random
0.31 1: 3.55
0.31 + 3.55
32.89
55.33
58.17
99.96
MrNets (Ours)
Global Expert
0.33 + 0.08
6.8e- + 7e-5
75.06
80.39
82.78
89.67
Hybrid Expert
0.33 0.08
6.8e-3 7e-5
75.78
95.33
95.06
93.81
Table 1: Algorithm performance on problems sets solvable by planner types. All prior methods use state-information and oracle collision checker while MæNets only needs point cloud
Evaluation Set
Training Set
MPNets-
Hybrid Expert Solvable (Ours)
MPNets [10]
-tylts-Style MPNets-Sttle
78.70
49.89
MrNets (Ours)
33.70
5.50
MPNets [10]
Hybrid Expert
88.90
65.28
MrNett (Ours)
Hybrid Expert
89.50
95.33
Table 2:
Success rates (%) of our method compared to Motion Planning Networks (MPNets) [10] trained and evaluated on different datasets
Hybrid End-Effector Space Planner Our Hybrid Planner struggles with a large proportion of problems solvable by the Global Planner. Yet, its solutions are both faster and smoother than the Global Planner. Surprisingly, MrNets trained with data from the expert outperformed the expert on the Global Planner-sol test set. We attribute this to two features: 1) we use strict rejection sampling to reduce erratic and divergent behavior in our expert dataset and train only on the filtered data and 2) our use of Hindsight Goal Revision to turn imperfect expert into perfect one.
Neural Motion Planning Motion Planning Networks (MPNets) [10] proposed a similar method for neural motion planning, but there are few key differences in both problem setup and system architecture. MPNets requires ground-truth collision checker to connect sparse waypoints, plans in configuration space, and is not reactive to changing conditions. the architecture, MPNets uses a trained neural sampler within a hierarchical bidirectional planner. The neural sampler is fully- connected network that accepts the start, goal, and a flattened representation of the obstacle points as inputs and outputs a sample. MPNets guarantees completeness by using traditional planner as fallback if the neural sampler fails produce valid plan.
In addition to our data, we generated a set of tabletop problems, which we call MPNets-Style, akin the Baxter experiments [10]. in order fairly compare the two methods. The results of this experiment can be seen in Table 2. MrNets requires large dataset that covers the space of test problems achieve compelling performance, while MPNets' utilization of a traditional planning system is much more effective with a small dataset or out-of- distriibitt problems. However, the MPNets architecture does not scale to more complex scenes, even with more data, as we show in Fig. 3. When trained and evaluated on the Hybrid Planner-solvable dataset, MPNets succeeds 65. 28% of the test set, whereas MrNets succeeds in 95.33%, thus decreasing the failure rate by 7X. Furthermore, Hybrid Planner performs similarly to uniform random sampler when both are embedded within the bidirectional MPNets planner.
we show in Table 1, using the MPNets neural sampler trained with the

---

% Within
% Env. Coll.
% Self Coll.
% Jnt Viol.
lem
Scm
15"
30
G. Fabrics [4]
8.61
0.11 0.11
0.44 0.25
69.89
75.17
83.44
85.11
STORM [12]
0.93
79.81
83.54
81.57
85.41
Netets (Ours)
Hybrid Expert
0.91 13.78
0.00
0.00
98.94
99.72
98.22
99.00
Global Expert
0.06
0.00
98.67
99.89
97.56
99.11
Table 3: Failure Modes on problems solvable by both the global and hybrid planners
Local Task Space Controllers Unlike planners, which succeed or fail in a binary fashion, local policies will produce individual actions that, when rolled out, may fail for various reasons. We break down the various failure modes across the set of problems solvable by both experts in Table 3.
STORM [12] and Geometric Fabrics [4] make local decisions that can lead them to diverge from the target in complex scenarios, such as cluttered environments or those with pockets. While STORM, Geometric Fabrics, and MrNets are all local policies. STORM and Geometric Fabrics rely human tuning to achieve strong performance. Prior environment knowledge alongside expert tuning can lead to phenomenal results, but these parameter values do not generalize. We used a single set of parameters across all test environments just as we used a single set of weights for Metets. MæNets encodes long-tern term planning information across wide variety of environments, which makes it less prone to local minima, especially in unseen environments.
On problems solvable by the Hybrid Planner, MTNets ties or outperforms these other methods across nearly all metries (see Appendix Table 1). On the set of problems solvable by the Global Planner, MeNNet target convergence rate is consistently higher, while its collision rate (11%) is worse than either STORM (1.94%) or Geometric Fabrics (7.83%) (see Appendix Table 2). Deteri- orating performance on out-of-distribution problems is a typical downside of supervised learning approach such as MrNets. However, this could be improved with a more robust expert, e,g, one with the consistency of our Hybrid Planner but the success rate of the Global Planner, with finetuning, or with DAgger [49].
5.2 Importance of the Expert Pipeline
We observed that the choice of the expert pipeline affects the performance of MrNets. We trained three policies: M*Nets-G with 6.54M demonstrations from the Global Planner. M*Nets-H with 3.27M demonstrations from the Hybrid Planner, and M*Nets-C with 3.27M demonstrations from each. M*Nets-C did not exhibit improved performance over either Net-Ht or Nrreet-G (see Appendix K for discussion). When evaluated on a test set of problems solvable by the Global Plan- ner, MrNets-G shows far better target convergence (97.9) ..4% vs. 87. 37.72%) compared to Mrrett-H but worse obstacle avoidance (21.94% collision rate vs. 11%). Nonetheless, MrNett-H is sig- nificantly better across all metrics when evaluated on problems solved by both experts as shown in Table 3. We hypothesize that an expert combining the properties of these two-the consistency of the Hybrid Planner and the generality of the Global Planner, would further improve MrNNet's performance. We refer to MrNets-HE H as MrNets throughout the rest of the paper.
5.3 Comparison to Methods With Partial Observations
In addition to demonstrating MmNets' performance on real robot system, we also compared McNett to the Global Planner (AIT* [42]) in a single-view depth camera setting in simulation. We evaluated on the test set of problems solvable by both the Global and Hybrid Planners. M*Nets only has a minor drop in success rate when using a partial point cloud vs. a full point cloud- from 95.06% 93.22 23. though the collision rate increases from 0.94% to 3.06% due to occlusions. For this experiment, we compared to the AIT component of our Global Planner alone to minimize false-positive solutions caused by the smoother's discrete collision checker (see discussion Sec- tion 4.2). We used voxel-based reconstruction akin to the standard perception pipeline packaged with Movelt [50]. In our implementation, a voxel filled only if 3D point is registered within it. On the same test set using the voxel representation, * produces plans with collisions on 16.41%

---

of problems. In this setting, MrNets's collision rate is over 5XX smaller than that of the Global Planner.
5.4 Ablations
100
We perform several ablations to justify our design decisions. All ablations were trained using the Hy- brid Planner dataset and evaluated on the Hybrid Planner-solvable set. More ablations and details can be found in Appendix K.
Succes Rate (9)
60
Nets Performance Scales with More Data As shown in Fig. 3, the performance of MrNets contin- to improve with data, although it saturates at 1.1M. Meanwhile, MPNets [10] has constant per- formance, demonstrating that our architecture is bet- ter able to scale with the data.
20
(MPNda)
Succem Raie (MtaNimo)
10
10a
10
Training Dataset Size
Figure 3: MrNets performance continues to increase with more training data, while MP- Nets performance stays relatively constant
Robot Point Representation Improves Perfor- mance Instead of representing the robot by con- figuration vector, insert the robot point cloud the specific configuration. Without this representation, the success rate decreases from 95.33% to 65.06%.
Hindsight Goal Revision Improves Convergence When trained without HGR, Le. with the plan- ner's original target given to the network, we see 58. 11% success rate vs. 95.33% when trained with HGR. In particular, only 60.28% of trajectories get within lem the target during evaluation.
Noise Injection Improves Robustness When we train MæNets without injecting noise into the input yr, the policy performance decreases by 10.72%.
5.5 Real Robot Evaluation
We deployed M*Nets on 7-DOF Franka Emika Panda robot with an extrinsically calibrated Intel Realsense L515 RGB-D camera mounted next to it. Depth measurements belonging to the robot are removed and re-inserted using a 3D model of the robot before inference with MrNets. We created qualitative open-loop demonstrations in static environments and closed-loop demonstrations in dynamic ones. Rollouts are between 2 and 80 time steps long depending on the control loop frequency. See Appendix M for system details. Results be viewed at https://mpinets, ittppiinetttttt.iiooo and the attached video. As can he seen, MeNees can achieve sim2real transfer on noisy real-world point clouds in unknown and changing scenes.
github.io
6 Limitations
While MrNeet can handle a large class of problems, they are ultimately limited by the quality of the expert supervisor and its need for a large, diverse dataset of training examples. Both generating the data and training McNets is computationally intensive, requiring access to equipment that is both economically and environmentally expensive. It will also struggle generalize to out-of- distribution settings typical of any supervised learning approach. See Appendix N for a deeper discussion on limitations and our plans for future work.
7 Conclusion
McNets is a class of end-to-end neural policy policies that learn to navigate to pose targets in task space while avoiding obstacles. MrNets show robust, reactive performance on real robot system using data from a single, static depth camera. We train MrNeet with what is, as far as we are aware, the largest existing dataset of end-to-end motion for a robotic manipulator. Our experiments show that when applied to appropriate problems, MrNets are significantly faster than global motion planner and more capable than prior neural planners and manually designed local control policies. Code and data are publicly available at https//mmpines.gitubbi..

---

Acknowledgments
We would like to thank the many people who have assisted in this research. In particular, we would like to thank Mike Skolones for supporting this research at NVIDIA, Balakumar Sundaralingam and Karl Van Wyk for their help in evaluating MrNets and benchmarking it against STORM and Geometric Fabrics respectively; Ankur Handa, Chris Xie, Arsalan Mousavian, Daniel Gordon, and Aaron Walsman for their ideas network architecture, 3D machine learning, and training: Nathan Ratliff and Chris Paxton for their help in shaping the idea early-on; Aditya Vamsikrishna, Rosario Scalise, Brian Hou, and Shohin Mukherjee for their help in exploring ideas for the expert pipeline, Jonathan Tremblay for his visualization expertise, Wei Chao and Yuxiang Yang for their help with using the Pybullet simulator, and Jennifer Mayer for editing the final paper.
References
[1] S. M. Vallee. Planning algorithms. 2006.
[2] N. D. Ratliff, M. Zucker, J. A. Bagnell, and S. Srinivasa. Chomp: Gradient optimization techniques for efficient motion planning. 2009 IEEE International Conference Robotics and Automation, pages 489,494,77 2009.
[3] J. Schulman, Y. Duan, J. Ho, A. X. Lee, I. Awwal. H. Bradlow, J. Pan, S. Patil, K. Goldberg, and P. Abbeel. Motion planning with sequential convex optimization and convex collision checking. The International Journal of Robotics Research, 33:1251 1270, 2014.
[4] K. V. Wyk, M. Xie, Li, M. A. Rana, Babich, B. N. Peele, Q. Wan, I. Akinola, B. Sun- daralingam, D. Fox, B. Boots, and N. D. Ratliff. Geometric fabries: Generalizing classical mechanics to capture the physics behavior. IEEE Robotics and Automation Letters, 7:3202- 3209, 2022.
[5] M. LaValle. Rapidly-exploring random trees: a new tool for path planning. The annual research report, 1998.
[6] E. Hart, N. Nilsson, and B. Raphael. formal basis for the heuristic determination of minimum cost paths. IEEE Trans. Syst. Sci. Cybern, 4:100-107, 1968.
[7] M. Likhachev, G. J. Gordon, and S. Thrun. Ara*: Anytime a with provable bounds sub- optimality. In NIPS, 2003.
[8] S. Karaman and E. Frazzoli. Sampling-based algorithms for optimal motion planning, The International Journal of Robotics Research, 30:846 894, 2011.
[9] M. P. Strub and J. D. Gammell. Advanced bit* (abit*): Sampling-based planning with ad- vanced graph-searde techniques. 2020 IEFE International Conference on Robotics and Au- tomation (ICRA), pages 130-136, 2020.
[10] A. H. Qureshi, M. Bency, and M. C. Yip. Motion planning networks. 2019 International Conference on Robotics and Automation (ICRA), pages 21 2118-2122 2019.
[11] B. Ichter, J. Harrison, and M. Pavone. Learning sampling distributions for robot motion plan- ning. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7087- 7094, 2018
[12] M. Bhardwaj, B. Sundaralingam, A. Mousavian, N. D. Ratliff, D. Fox, F. Ramos, and B. Boots. STORM: An integrated framework for fast joint-space model-predietive control for reactive manipulation. 2021.
[13] J. C. Kew, B. Ichter, M. Bandari, T.-W. E. Lee, and A. Faust. Neural collision clearance estimator for batched motion planning. arXiv preprint arXiv:1910.05917 2019.
[14] R. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, Shotton, Hodges, and A. Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking- 201 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2011.

---

[15] J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continu- ous signed distance functions for shape representation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[16] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[17] M. Danielczuk, A. Mousavian, C. Eppner, and D. Fox. Object rearrangement using learned implicit collision functions. 2021 IEEE International Conference Robotics and Automation (ICRA), pages 6010-6017, 2021.
[18) T. Jurgenson and A. Tamar. Harnessing Reinforcement Learning for Neural Motion Planning. In Robotics Science and Systems, 2019.
[19] A. Tamar, Y. Wu, G. Thomas, S. Levine, and P. Abbeel. Value iteration networks. In Neural Information Processing Systems, 2016.
[20] R. A. M. Strudel, G. Pinel, Carpentier, J.-P. Laumond, I. Laptev, and C. Schmid. Learning obstacle representations for neural motion planning. In CoRL, 2020.
[21] M. Likhachev, Ferguson, G. Gordon, Stentz, and S. Thrun. Anytime dynamic a*: An anytime, replanning algorithm. In ICAPS, 2005.
[22] D. Gammell, S. S. Srinivasa, and T. D. Barfoot. Batch informed trees (bit*): Sampling-base optimal planning via the heuristically guided search of implicit random geometric graphs 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 3067-3074, 2015.
[23] N. D. Ratliff, M. Toussaint, and S. Schaal. Understanding the geometry of workspace obstacles in motion optimization. 2015 IEEE International Conference on Robotics and Automation (ICRA). pages 4202 4209, 2015.
[24] O. Khatib. Real-time obstacle avoidance for manipulators and mobile robots. In Autonomous robot vehicles, pages 396-404. Springer, 1986.
[25] N. D. Ratliff, J. Issac, and D. Kappler. Riemannian motion policies. ArXiv, abs/ 18.8544, 2018.
[26] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective on imitation learning. Found. Trends Robotics, 7:1-179, 2018.
[27] S. J. Russell. Learning agents for uncertain environments (extended abstract). In COLT" 98, 1998.
[28] A. Y. Ng and S. J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning. ICML '00, page San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.
[29] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforce- ment learning. In AAAI, 2008.
[30] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In NIPS, 1988.
[31] M. Bain and C. Sammut, A framework for behavioural cloning. In Machine Intelligence 15, 1995.
[32] R. Kumar, A. Mandalika, Choudhury, and S. Srinivasa. Lego; Leveraging experience in roadmap generation for sampling- planning. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1488-1495, 2019.
[33] C. Zhang, J. Huh, and D. D. Lee. Learning implicit sampling distributions for motion planning. 2018 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3654-3661, 2018.
10

---

[34] C. Chamzas, Z. K. Kingston, C. Quintero-Peñaa, A. Shrivastava, and L. E. Kavraki. Learning sampling distributions using local 3d workspace decompositions for motion planning in high dimensions. 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 1283-1289, 2021.
[35] D. S. Chaplot, D. Pathak, and J. Malik. Differentiable spatial planning using transformers. In Internal Conference Machine Learning. 2021.
[36] B. Eysenbach, R. Salakhutdinov, and S. Levine. C-learnigg: Learning achieve goals via recursive. In International Conference on Learning Representations, 2021.
[37] C. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets a metric space. In NIPS, 2017.
[38] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In International Conference on Computer Vision (ICCV), 2019.
[39] A. Murali, A. Mousavian, C. Eppner, C. Paxton, and D. Fox. 6-dof grasping for target- object manipulation in clutter. In IEEE International Conference on Robotics and Automation (ICRA), 2020.
[40] C. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning on point for 3d classifica- tion and segmentation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 77-85, 2017.
[41] A. Fishman, C. Paxton, W. Yang, D. Fox, B. Boots, and N. D. Ratliff. Collaborative interaction models for optimized human-robot teamwork. 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11221-11222 2020.
[42] M. P. Strub and J. D. Gammell. Adaptively informed trees (ait*): Fast asymptotically optimal path planning through adaptive heuristics. 2020 IEEE International Conference Robotics and Automation (ICRA), pages 3191-3198, 2020.
[43] R. Diankov. Automated Construction of Robotic Manipulation Programs. PhD the- sis, Carnegie Mellon University, Roboties Institute, August 2010. URL http://www. programmingvision htt.m//migniiii.o...oooe.eeeeeee
cerssssspfff.. pdf.
[44] K. K. Hauser and V. Ng- Di-Thow-Hiing Fast smoothing of manipulator trajectories using op- timal boulded-accollrration shortcuts. 2010 IEEE International Conference on Robotics and Automation, pages 2493-2498, 2010.
[45] M. Andrychowicz. D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, Abbeel, and Zaremba. Hindsight experience replay. In NIPS, 2017.
[46] Wunsch, S. Winkler, and G. Hirzinger. Real-time pose estimation of 3d objects from cam- era images using neural networks. Proceedings of International Conference on Roboties and Automation, 4:3232-3237 vol.4, 1997.
[47] S. Balasubramanian, A. Melendez-Calderoo A. Roby-Brami, and E. Burdet. On the analysis of movement smoothness. Journal of NeuroEngineerrng and Rehabilitation, 12, 2015.
[48] M. Mukadam, Dong, X. Yan, F. Dellaert, and Boots. Continuous-ting Gaussian process motion planning via probabilistic inference. volume 37, pages 1319-1340, 2018.
[49] S. Ross, G. J. Gordon, and J. A. Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In AISTATS, 2011.
[50] S. Chitta, I. Sucan, and S. Cousins. Moveit![ros topics]. IEEE Robotics & Autommatio Maga- zine, 19(1):18-19 2012.


Reviews:
[{"review_id": "zUHx_w-470r", "reviewer": "Reviewer_V556", "summary": "This paper presents an end-to-end neural motion policy network that learns to generate collision-free trajectories from a point cloud of the scene, the goal state and current joint configuration. It is trained on a large-scale dataset of millions of sample motion planning problems from hundreds of thousands environments. The approach, referred to as M$\\pi$Nets, is directly comparable to the MPNets [11] algorithm, showcasing improved performance when sufficient data is available without requiring collision-checking and being able to react in dynamic environments. The data used to train an M$\\pi$Net is generated from an expert hybrid planner that leverages the strengths of global and local planners. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is well-written and enjoyable to read. \n- It proposes an interesting solution to difficult motion planning.\n- The fact that it is trained on raw point cloud observations makes the approach amenable to sim2real without any additional modifications.\n- The evaluation of accuracy and performance of M$\\pi$Nets vs. baselines and other SOTA is sufficient and rigorous (except for computation times – discusses in weaknesses)\n- Simulated experiments are impressive.\n- Inclusion of failures in the video is appreciated. \n\n**Weaknesses**\n- While the comparisons and metrics used to evaluate M$\\pi$Nets do a very good job at analyzing its performance based on task/collision-avoidance success, there is no mention about computation time either at training or test time. Only until we go to the appendix we see that M$\\pi$Net takes 1 weeks to train with 8 NVIDIA Tesla GPUs. This seems like a very important detail to leave out from the main paper. Furthermore, I would have expected to see a comparison of training/testing time vs. MPNets. How much compute is needed to run the network at test time? Do you need the 8 NVIDIA Tesla GPUs? When proposing an algorithm that requires very heavy computation, one must consider that this might not be accessible or even feasible for certain scenarios or circumstances – hence, the hardware requirements must be transparent.\n- Throughout the text it is noted that low performance due to distribution shift could be alleviated by combining the global and local planners. It would be great to see in the paper an explanation as to why this is difficult. Couldn’t you combine the successful demonstrations from both the global and local experts? I assume this might introduce inconsistencies in the data and instead of improving the overall performance it would deteriorate it. Is this the case? It would be great to include such comparison or some extended analysis on this and some suggestions as to how these experts could be combined. Have the authors thought of employing an expert combination strategy like Bayesian Model Combination as done here for an RL setting: https://dl.acm.org/doi/10.5555/3327546.3327623 \n- In Table 2, the success rate for M$\\pi$Nets tested on the solvable problems seems extremely low 5.5%, could this result be elaborated? What is so fundamentally different between the Hybrid Expert datasets and the MPNets-Styles?\n- The two control methods should be mentioned in the main text (frequency for closed-loop control) and further details remaining in appendix.  \n- It seems that the points clouds are always uniformly sampled both at training and test time? Is this correct? What is the performance with truly noisy input point clouds?\n- The catastrophic and erratic behaviors that are due to distribution shift make it difficult to see this learned end-to-end network working on real-world scenarios. Apart from using/generating more data or allowing corrections to the learned model like DAGGER, there must be a fundamental safety component that allows the robot to at minimum not collide with the environment. Have the authors thought about introducing such a mechanism to the framework?  It would be interesting to see this discussion. \n\n\n**Further Improvements/Suggestions**\n\n- To learn a robust policy that is “consistent” with the expert demonstrations the authors might want to consider augmenting the imitation loss with some notions of stability as in:\nhttps://arxiv.org/pdf/2205.14812.pdf\n- It would be interesting to include more metrics to evaluate the distinct behaviors of each motion planner, like: obstacle clearance, near-obstacle velocity and motion legibility. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents an interesting solution to a classical problem in robots - efficient collision free motion planning. This is done by an end–to-end motion policy network that takes as input a point cloud of the scene, the target and the joint configuration of the robot and is learned by a large scale dataset of expert demonstrations provided by planners. While the idea of learning from expert planners is not novel, the performance and range of environments that this motion policy network can handle is unprecedented. The paper needs to be more forthcoming about the computation power necessary to train and run such networks. \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Comments listed in the weaknesses section should be addressed. ", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "nUqCb-3OeH-", "reviewer": "Reviewer_3osu", "summary": "This paper primarily introduces a novel neural policy architecture - named M$\\pi$nets - that can be used to generate robotic motions that achieve a particular goal configuration from a single depth image. To complement this, the authors create a novel extremely-large dataset of motion-planning problems and solutions on which they train their approach. Across extensive experiments, they show that M$\\pi$nets not only outperform existing approaches in simulation, but also transfer well to novel tasks with noisy point clouds on real robotic hardware.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Simplicity of the method: The presented M$\\pi$nets architecture and loss functions used for training are rather simple, especially compared to certain baselines, yet perform remarkably well. This simplicity makes the method easy-to-understand and also provides good intuition for why it works.\n- Extensive experimental evaluation: The authors' experimental evaluation is extremely thorough and extensive: there is not a single additional result I can think of that I would like to see.\n- Well-written paper: The paper's writing and figures are extremely clear and easy-to-follow. Additionally, most important details are provided: I feel that a reader could re-implement the method after carefully studying the paper.\n\nWeaknesses:\nNo major weaknesses I could think of.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I believe this is a strong paper that not only makes several useful contributions to the robotics community, but is also well-presented and easy to follow. In particular, the authors' method and novel dataset offer a new tool towards deploying motion-planning in real-world situations, which has been a goal of the robotics community for several decades. The experimental evaluation is thorough and convincing. I don't see any major flaws that could cause me to recommend rejection.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "**Questions**\n(Note that these are not issues I have with the paper, but more of a curiosity/question. If possible, it would be great if you could include additional details answering this in either the main paper or supplementary materials.)\n- Encoding the target in the point cloud: Is there some reason/intuition for why you decided to encode the target gripper position as part of the point cloud? It seems more straightforward to simply specify the target as a configuration ($q$) vector. Additionally, one of your ablations seems to show that if you remove $q_{t}^{||\\cdot||}$ and instead supply this as part of the point cloud, performance degrades substantially. Wouldn't this suggest you can improve performance by also providing the target as an explicit vector instead of as part of the point cloud?\n\n"}}, {"review_id": "T9BPBrNVfUM", "reviewer": "Reviewer_oeVp", "summary": "The authors propose Motion Policy Networks, a learned policy trained on data from classical motion planning solutions. The policy takes as input a configuration space state and a point cloud and outputs configuration space actions. The authors propose additional losses based on motion planning and collect data from motion planning solutions. M\\piNets then train a policy and roll out the actions for solve planning problems. This approach is ablated over each loss, data source, and compared to several motion planning techniques and shown to scale well with data and achieve high performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clear and the experiments are well done. The point cloud representation is nice and general, especially compared to representations used in many motion planning and learning + motion planning algorithms. The experimental environments are nice and reasonably complex. The figures are clear as well. I think Figure 3 is informative and clear, that policies have more performance as data increases, whereas structured approaches like MP-Nets outperform in a limited data regime. The ablations are well done and thorough (ablating the loss, data sources, observation, etc) – and informative.\n\nThe primary weakness is the framing of the approach. This approach at its core is behavioral cloning in configuration space with additional loses and data training sources. But the authors discuss in detail the motion planning literation as well as learning for motion planning, and ablate and compare to baselines in those fields thoroughly. E.g., they compare to MP-net, which they rightly note requires collision checking, it also has guarantees of falling back on motion planning. \n\nInstead the authors should be comparing to baselines for learned robotic policies and should be framing the paper accordingly. In framing this way, the authors can discuss data sources more clearly. Questions such as BC data if the data is smooth, noisy, has global coverage are interesting and useful for learning policies. The authors can also show how task-specific losses for motion planning help and hurt. \n\nMinor comments:\n- Can you clarify again why the global planner fails on hybrid problems, when I thought hybrid problems require a global solution to then be smoothed.\n- The authors should show an image of the planning problems in the main body of the paper.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors have written a thorough and clear paper, but it is framed around motion planning when the algorithm is an imitation learned policy with ablations over data sources and task-specific losses. The experiments are thorough and indeed an interesting comparison with motion planning, but it would be useful to better understand how it fits within the imitation learning literature. It would also make comparisons for when learned policies outperform motion planning and when they can learn from motion planning more clear.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See above, primarily with respect to framing.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "OkffhBGPI-3", "reviewer": "Reviewer_S422", "summary": "The paper proposes a neural motion planning learner, which takes a point cloud and a robot configuration as input and outputs the next configuration. The learned model can handle unknown, partially observable, and dynamic environments. Its outperforming performance over various baselines empirically proves its applicability to real-time practical scenarios.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The performance of the proposed framework is impressive, especially in dealing with dynamic environments.\n- Evaluation is exhaustive as various metrics are used, and multiple baselines are compared.\n\nWeaknesses (details are included in the Summary Of Recommendation and Issues sections):\n- Relatively simple problems are evaluated. Problems having narrow passages, the most notorious challenge in motion planning, are not sufficiently assessed, in which a learning-based method is expected to work poorly.\n- Some points (e.g. task space and the notion of consistency) made in the paper are vague, thus arguable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well-organized, and its presentation is clear. Exhaustive evaluations are impressive, and the video demonstration clearly conveys the method's performance. I think the paper can further be improved by addressing the weakness points made above. In particular, whether to be able to handle more intricate problems having narrow passages is of significant concern.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Major comments:\n- Harder problems (narrow passages): As pointed out above, most of the problems shown in the paper and the video look relatively simple (wide passages in the configuration space). The only hard one is the limitation scenario in the video. Since the proposed framework is learning-based, its fast computation is not surprising compared to classical planning algorithms. I think a more rigorous analysis of whether challenging problems (e.g. arm going through multiple objects in between) can or cannot be solved is needed to understand how much completeness is sacrificed.\n- Consistency (and task space): The authors emphasize the importance of the notion of consistency in many places, but its definition is vague and rather handwavy. It is defined in the paper as: \"consistency describes how the motion would change in response to a perturbation in any dimension of the planning problem.\" Is it a subjective definition? Otherwise, please redefine this notion precisely. This confusion incurs sequential confusion as follows. I couldn't understand why learning in the task space is more beneficial than learning in the configuration space, which has some connection to consistency. Also, the motivation of hybrid planners is unclear. Since this is the primary motivation of the proposed framework, clarifying this point is critical.\n- Being able to handle partial observability: The model is trained with fully-observable data, but it is unclear how the ability to handle partial observability at test time emerges.\n- Hybrid planner: Because of the confusion about consistency, the fundamental difference between global planner and hybrid planner is vague, making the rest of the paper hard to understand. Moreover, in Section 5, what would be the fundamental differences between problems solvable by either one? Please clarify it. The word \"hybrid\" comes from the fact that it is a combination of the global planner and the local planner?\n\n\nMinor comments:\n- In the contribution bullet points, \"46% better\": The word \"better\" is not well-defined. Is it referring to efficiency or the success rate? Please clarify it.\n- In the introduction, \"enforce strong assumptions about ...\": Please briefly explain the assumptions.\n- In Section 2, global planning: The authors state that \"optimal paths in configuration space can lead to unintuitive paths in a task space\", which may be related to the notion of consistency. What would be intuitive paths in a task space? Distinguishing intuitive paths from unintuitive ones seems arbitrary and subjective. Please provide one example.\n- In Section 3.3, \"task spaces on the surface of the robot\": I couldn't understand the relationship between the task space and the surface of the robot.\n- In Section 3.4: Do the closed-loop rollouts mean that the loss is computed from not a single configuration but an entire trajectory?\n- In Section 3.4, \"We also sample novel scene point clouds on-the-fly during training\": I couldn't understand this sentence. Does this mean one problem could have entirely different point clouds from the current scene? I am not sure how learning is possible from it.\n- In Section 4.2, global planner: The authors state that the global planner fails if the smoother produces collisions. I am questionable if it is a major and fair cause of failure because it is just a wrong hyperparameter tuning.\n- In Section 4.2, hybrid planner: Please clarify \"create a consistent configuration space velocity profile\", which is hard to understand.\n- In Table 3, I couldn't understand why there is such a large gap between hybrid expert and global expert on the environment collision (i.e. 0.94 vs. 13.78), which is related to one of the above comments. Is it still happening even after a careful hyperparameter tuning of the global planner?\n- In Section 5.1, the results of the hybrid planner: There are 11%, 2.06%, and 7.94% written in the text, but they don't correspond to any numbers in Table 3. Are they separate results?\n- In Section 5.3: 16.41% depends on how the authors considered unknown regions of problems in training. Please explain the method briefly.\n- Typo in Section 7: From \"end-to-end neural molicy policies\", remove \"molicy\"."}}, {"review_id": "LTruoLnAeQl", "reviewer": "Reviewer_Eji4", "summary": "The paper presents an end-to-end deep learning based motion planner that is designed for use with manipulator arms and point cloud representations of obstacles. The goal of the work is to increase planning speed versus previous methods. For example, as compared to sampling based motion planning, graph search, and other forms of optimization. The algorithm is trained using millions of expert trajectories and two expert systems (one based on a sampling based motion planing algorithm, and another based on a hybrid algorithm). Extensive experiments and ablation studies are performed in simulation and there is also a statement that the work has been evaluated on a real robot. \n\n\n", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths of the paper are as follows:\n1) The training set is very large \"nearly 300x larger than previous work\" according to the paper. \n\n2) Many different metrics were collected including: success rate, time, rollout target error, collision rat, and smoothness.\n\n3) The number of simulations performed is extensive.\n\n4) The experiments show that using the learned model instead of planning from scratch reduces runtime and increase smoothness (though the model also has decreased performance with respect to other metrics such as success rate). \n\n\nThe main weaknesses of the paper are as follows:\n1) There is little discussion of the relationship between the training and the test sets. Therefore, it is impossible for this reviewer to evaluate if the test set was chosen fairly, and hence, the quality or generality of the experimental results cannot be assessed. \n\n2) There does not appear to be experimental data/results regarding the performance of the algorithm in dynamic scenarios, despite this being cited as a benefit of the proposed method in the paper, nor when a solution is asked to solve a problem in which a solution does not exist. Testing in cases that do not have solutions is as important as testing in cases that do have solutions.\n\n3) The discussion of graph search and sampling based motion planning algorithms contains many statements that are not true (see A-C below).\n\n4) Despite claims of sim-to-real transfer, the hardware experiments are anecdotal and not empirical. \n\n5) The  quality or generality of the method with respect to training set size and scope should be discussed in the limitations sections. For example, how close does the training and test set need to be to get decent performance in practice. How  well does training on simulation generalize to testing on hardware?\n\n\n-- incorrect claims about sampling based motion planning and graph search --\n\nA) The paper states, \"Global planners such as RRT [1] optimize for speed and completeness but fall short on optimality.\" This quote communicates a fundamental misunderstanding about the goals of feasible sampling based motion planning. RRT and other *feasible* sampling based motion planning algorithms are designed to answer the question: \"is it possible to reach the goal from the start?\" Answering this question in the affirmative is possible given any valid path, and so all valid paths are considered equally \"good\" by RRT.  So, saying feasible algorithms  \"fall short\" with respect to optimal is incorrect.  Using RRT when one cares about optimality is simply using the wrong tool for the job.\n\nB) The paper states, \"Search-based planning algorithms ... are slow but find least-cost paths... Sampling based planners are typically much faster...\" Whether either class of algorithms is \"fast\" or \"slow\" depends on how they are used. One can trade speed for better resolution. Moreover, graph search on n nodes is typically many orders of magnitude faster than using sampling based motion planning to create a graph of n nodes from scratch. So, if we fix a lattice graph of a certain resolution then graph-search will require a runtime that scales to the power of the number of C-Space dimensions; on the other hand, if we fix the number of nodes and neighbors then the runtime will remain constant vs. the number of dimensions but the resolution (of nodes per unit hyper-volume) will decrease. \n\nC) The paper states, \"The Global Planner {AIT*} is theoretically complete and optimal, but in practice, we observed several common failures types\" and \"Global Planner is theoretically complete.\" This is untrue. AIT* is neither complete nor optimal.  Rather, AIT*, as with all almost-surely asymptotically optimal sampling-based planing algorithms, is *probabilistically complete* and *asymptotically optimal*. Inherent in these definitions are the facts that more planning time will yield better results, on average, and that there is a stochastic nature to the algorithms' performance. It may also be the case that no valid solution exits (in which case the algorithm will not return a solution). \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The idea is interesting and within the scope of the conference, yet there are also a number of errors in the paper regarding previous work, missing details regarding the training and testing process, and claims that are not backed up by experimental results. The main contribution of this paper is the proof of concept demonstration in simulation that end-to-end manipulator planning is possible given point cloud data. While many experiments are run, it is hard to assess the results because there is no indication of how closely the test set resembles the training set. There do not appear to be any empirical results from experiments run in dynamic environments, nor in cases where solutions do not exist, nor on robotic hardware. While the core idea is interesting and a good goal, this version of the paper leaves too much room for improvement.  \n\n\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1) Right now the lack of information regarding how training and test sets related makes it hard to understand how to apply this method because there is not evidence of how much a trained model is able to generalize about what it has learned. For example, does training on a table top produce a model that can be used to plan on a counter or on the floor? How close in shape do the objects in the test cases need to be to the objects used in the training examples? Adding such discussion would improve the paper.  Even better would be the addition of empirical assessment of these things.  \n\n2) Deployment in dynamic environments is cited as a motivation for the work multiple times. However, the paper does not appear to contain the results of experiments from dynamic environments. Performing such experiments and reporting how performance changes (e.g., success rate) as a function of, e.g., the speed of environmental changes would improve the paper. Otherwise, claims of deployment in in a dynamic environment should be removed from the paper. \n\n3) Considering cases where solutions do not exist is just as important as considering cases where solutions do exist. Testing what happens when no solution exists will improve the paper (how often is there a false positive path result that leads to a collision?). \n\n4) The discussion of sampling based motion planning algorithms contains many statements that are not true. See (A-C) in the \"Strengths and weaknesses\" section.  This language should be changed to include more nuanced discussion of A-C.\n\n5) The paper states that \"MπNets is only trained for single-step prediction, but we use it for closed-loop rollout.\" Yet, the paper also states that Geometric Fabrics [9] and STORM [10] that \"deploy reactive local policies and assume that local decisions will lead to globally acceptable paths\" but have worse performance in practice. All three (MπNets, Geometric Fabrics, and STORM) appear to be making local decisions (or single-step prediction) and have an assumption that stitching them into global paths will work. The paper can be improved by adding additional information/discussion about why performance differs between these methods\n\n6) Despite claims of sim-to-real transfer, the hardware experiments are anecdotal and not empirical. Adding empirical hardware experiments would improve the paper by quantitatively demonstrating the sim-to-real transfer. This would also help to alleviate worries of training vs. test sets (that is, if the training set is from simulation and the test set is from real hardware).\n\n7) The  quality or generality of the method with respect to training set size and scope should be discussed in the limitations sections. For example, how close does the training and test set need to be to get decent performance in practice. How  well does training on simulation data generalize to testing on hardware?\n"}}]

Author rebuttal:

Decision: Accept (poster)