Predicting Stable Configurations for Semantic Placement of Novel Objects
Authors: Chris Paxton
Chris Xie
Tucker Hermans
Dieter Fox
Venue: CoRL2021 Poster (2021)
Keywords: Deep learning for robotic manipulation
learning for motion planning
semantic manipulation

Abstract: Human environments contain numerous objects configured in a variety of arrangements. Our goal is to enable robots to repose previously unseen objects according to learned semantic relationships in novel environments. We break this problem down into two parts: (1) finding physically valid locations for the objects and (2) determining if those poses satisfy learned, high-level semantic relationships.
We build our models and training from the ground up to be tightly integrated with our proposed planning algorithm for semantic placement of unknown objects. We train our models purely in simulation, with no fine-tuning needed for use in the real world.
Our approach enables motion planning for semantic rearrangement of unknown objects in scenes with varying geometry from only RGB-D sensing. Our experiments through a set of simulated ablations demonstrate that using a relational classifier alone is not sufficient for reliable planning. We further demonstrate the ability of our planner to generate and execute diverse manipulation plans through a set of real-world experiments with a variety of objects.


Predicting Stable Configurations for Semantic Placement of Novel Objects
Chris Paxton', Chris Xie?, Tucker Hermans Dieter Fox IVVIDI, Unniveriity of Washington, $University of Utah
Abstract: Human environments contain numerous objects configured in variety of arrangements. Our goal is to enable robots to repose previously unseen objects according to learned semantic relationships in novel environments. We break this problem down into two parts: (1) finding physically valid locations for the objects and (2) determining if those poses satisfy learned, high-level semantic relation- ships. We build our models and training from the ground up to be tightly inte- grated with our proposed planning algorithm for semantic placement of unknown objects. We train our models purely in simulation, with no fine-tuning needed for use in the real world. Our approach enables motion planning for semantic rear- rangement of unknown objects in scenes with varying geometry from only RGB- sensing. Our experiments through set simulated ablations demonstrate that using relational classifier alone is not sufficient for reliable planning. We further demonstrate the ability of our planner to generate and execute diverse manipula- tion plans through a set of real-world experiments with variety of objects.
Keywords: Deep learning for manipulation, learning for motion plan- ning, semantic manipulation
"Place the mug front of blue pitcher."
"Place the pitcher of mustard."
"Place the mustard behind the cookie box."
"Place the mug on top of the box."
Figure 1: Planning to achieve different predicate relationships in the real world, given only segmented point clouds of the scene. The planner computes final placement positions and associated kinematic poses, which lets the robot generate motion plan to place these objects at different locations in the real world.
Introduction & Motivation
Many robotics tasks human environments involve reasoning about the relationships between dif- ferent objects and their locations in a particular environment. Imagine robot tasked with pouring a cup of coffee, it must reason about relative position of the cup to the coffee pot, such that pours the coffee into the cup and not onto the counter or floor. For robot attempting to brew the coffee prior to pouring, many more multi-object relations must be reasoned about for the tasks of retrieving the necessary equipment from storage, scooping and pouring beans grounds, and filling any required water vessels.
Classically, researchers investigating these sorts of multi-object planning and manipulation prob- lems assume full knowledge of the objects and their poses the scene [1]. However, for robots acting in homes and other world environments, it is unreasonable to assume that a deployed robot will have knowledge and accurate pose estimation all objects in the environment. Further- more, simple hand-engineered classifiers for determining logical predicates often exhibit unintended
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

behavior when deployed on real-world systems [2]. For these reasons, rearrangement has recently been identified as major challenge for robotics [3].
Given these challenges, its natural to examine the use of learning to improve task and motion plan- ning with real sensing [4, 5, 6, 7, 8, 9]. However, previous methods fail to solve the full problem of unknown object rearrangement with physical robots, Some only operate on known objects [6. 9], others ignore or significantly restrict the space of robot control [4, 5] or relations [7, 8], while still others make assume an explicit goal configuration is given [5], An alternative approach to solve com- plex manipulation tasks relies on learning model-free neural net policies instead of explicit models of conditions and effects [10, 1 11]. Such methods have so far failed to show the level of generalization across objects and environments capable by modern task and motion planners.
In this paper, we focus on a critical subtask of rearrangement planning-semanti placement-where the robot must perform pick and place operation to move an object into stable configuration that satisfies a desired set of semantic relations. This allows us to focus on three specific objectives not addressed in previous work on learning for task and motion planning. First, we examine the problem of planning and controlling manipulation behaviors to change inter-object relations of potentially previously unseen objects using RGB-D sensing. Second, we aim to learn the necessary relations from data to avoid the bias introduced from hand-engineered classifiers. Finally, we must learn what realistic scenes look like-so so that the robot can ensure that it places objects in reasonable locations which are stable and free of undesired collisions.
Our work relies the ability to infer inter-object relational predicates between objects pro- posed from an RGB-D instance segmentation, e.g. [12]. Semantic relations serve an important role in instructing robots [13]. As such, researchers have examined visual prediction of spatial rela- tions [14, 15, 16, 17] including inference of support relations [18, 19]. While these methods can be sophisticated incorporating language [20, 21, 16] scene graph information [22, 23, 17]; none have demonstrated the ability to integrate prediction with robot planning and control to satisfy new relations between unknown objects. Indeed, we show semantic prediction alone is insufficient for reliably planning successful semantic rearrangement.
In addition to this semantic prediction, need to identify physically realistic poses where these objects can be placed. Stable placement prediction has also received attention from robotics re- searchers [24, 25, 26, 27]; however, learning- approaches [24, 27] train directly maximize stability of object placement. In contrast our approach simply distinguishes realistic configurations from invalid ones, which allows us to learn general-purpose scene-realism discriminator which can capture wide distributions over realistic poses in 3D space. This provides the benefit of a more general model for use in arrangement planning, while giving up somewhat the level of precision seen in some placement-specific methods [24, 26]. In addition, it means we can use simulation- based data to train both our relational prediction and scene discriminator directly on raw point cloud and associated segmentation masks, and transfer to the real world.
We embedded our learned relational predictor and scene discriminator within sampling-based based planning framework to change relations in the scene in a goal directed way. By planning directly over changes in segment pose in point cloud, we can decouple the goal generation problem from the robot control problem, allowing us to leverage modell-based state-of-the-an motion planning and grasp prediction algorithms [28, 29] to perform the necessary manipulation. We further accelerate our planner by learning an object pose sampler conditioned on desired relations to initialize the optimization, similar to the grasp planning approach in [28]. Production of these goal states can then be used for semantically-deiin defined placement tasks.
We highlight the advantages of our approach over variety of baselines and ablations of our full method. Critically, we demonstrate that both the relational classifier and scene discriminator are nec- essary for reliably generating successful plans. We then demonstrate our approach in the real world using a Franka robot (Fig. 1). Our experiments constitute the first physical-robot demonstration that combine learned models for inter-object relations and stability estimation enabling rearrangement of novel objects. Crucially, we show that the combination of relationship classifier and scene discrimi- nator allows to plan placements for a variety of relationships in cluttered scenes. In addition, our models are trained entirely in simulation with no need for real-world fine tuning.
2

---

2 Methods
Given a single view of a scene containing objects for which our robot potentially has no previous experience, we wish for the robot to rearrange the scene to satisfy some new set of logical constraints. For example the robot may be tasked to move the query object to be on the far right side of anchor object j or to be stacked on top of object k. Each individual relationship between i and is referred to as a predicate Piji we can describe multiple logical relationships as the vector Piz.
We assume we are given a partial-view point cloud Z with segment labels for each point to identify the different objects, Given this point cloud and set of logical predicates describing the desired relationships Ptz. the robot must find a pose offset 8 (3D translation and planar rotation) for object i that satisfies Pij and is additionally a stable, physically valid placement pose in the environment.
Thus there are two key components in our rearrangement and placement planning approach: pre- dicting which poses objects can be physically placed and predicting which poses satisfy the given high-level instructions. We formalize predicate planning as the following problem:
argmin
c(o) = , f(a, @ 6, z') 112 1 (2) @ &6,z)) - Aill2
(1)
subject to
T(Z,z, z. + 06) = '2'
(2)
f(a @ 6, 2) > >ef
(3)
(a 2 8,zz) [ris] > tp
(4)
[I(;; @ 6) =
(5)
where F; and 2y are the object point clouds for objects i and j. respectively, and @ denotes the application of the 3D translation and planar rotation. At the heart of our planning cost, Eq. 1, are two models. The first f(; @ 8, Z') is neural net trained to determine if the resulting scene is physically realistic and stable. The second term defines the cost associated with matching the set of target predicates, where Pp (2, ( 6,2;) estimates the set of predicate relationships between the transformed point cloud z; @ and zy. This cost implies maximizing the likelihood that the resulting scene both realistic and satisfies the desired predicates.
In addition to the cost function, we put minimum bound constraints on the physical feasibility (Eq. 3) and predicate probabilities (Eq. 4). We use pij as index in Eq. 4 to extract the subset of predicted relations that must be satisfied. This ensures we never attempt to plan to a scene configuration with low probability of success, even if it defines a local optimum of the objective. Eq. 2 models the transition of applying offset ô to I; in the observed scene 2 to generate the resulting scene Z', which we evaluate in the cost and other constraints. Finally Eq. ensures that sampled object offsets are visible in the camera view of the robot, since our cost and constraint evaluations would be ill- defined otherwise. We describe the details of these models and their construction in Sec. 2.1 and give detailed description of our data generation process for training in Section 2.3.
We use a variant of the cross-entropy method (CEM) [30] to solve this constrained optimization problem. CEM has previously been applied to robot motion planning [30], including semantic mo- tion planning from learned models [31]. We provide further details of our planner in Section 2.2.
2.1 Rearrangement Planner Models and Training
We train multiple neural networks to compute the values needed to instantiate our planning problem defined by Equation 1: the current set of predicates, and the discriminator score which describes whether or not a particular set of object points æ defines a realistic configuration in the scene.
Object and scene encoders The core piece of the model is a PointNet++ [32] encoder which extracts a dower-dimensional object representation h. Each network takes two objects i and j represented as point clouds I; and zzj. as well as the scene point cloud Z, as input. Given an observation point cloud I, we learn a mapping e(x) -> h for the objects, in order to get two latent representations h; and hy for the query and anchor objects. The object encoder is Pointnet++ model with three set abstraction layers; see the supplemental material for further details.
We train a separate scene encoder ez ez(#z) (xz) to capture the objects' relation to other scene geometry when predicting where can placed. This outputs liz encoding scene-specific information, where zz is centered around the anchor point 0j the centroid of zz. Points are sampled in a radius of
3

---

Pointnet Encoders
i
P p
p
Scene Discriminator
hy
π
Placement Planner
Z
Motion Planner
Ag
Goal specification
Figure System for rearrangement unknown objects according to spatial relations, where we wish to move a query object i (in this case a red mug). We encode objects with Pointnet++ to predict predicates, based on the region around an "anchoring" object, j (the macaroni box) to which the query is relative. Query locations are sampled based
learned prior v, and are classified by the or unrealistic. This is used as a part of an optimization algorithm to find a stable, kinematically feasible query pose so that we can place the query object in new location.
discriminator network f as either realistic
r = 0.5m around oj. We choose this representation to define features from the fixed perspective of the anchoring object j.
Relation predictor The predicate classifier network pp x.y) + P. estimates which predicates true for a pair of object point clouds. It uses the representations from the object encoder, e(x;) h; and e(r;) = 12,. for the query and anchor objects and j. respectively. These are passed into an MLP which predicts vector of length N Npredicates- When used as part of the planning algorithm, we ignore predicted predicates that are not part of the goal specification.
Pose prior In addition, learn prior distribution over possible poses where the object might satisfy the predicates, relative to the anchoring object. We implement this prior T zy. Z, pby) + {ak, Mr: T k=1:K
Mixture Density Network (MDN) [33], which predicts the parameters of a Gaussian mixture model with K components. This GMM distribution defines the probability of specific pose offsets 8 with respect to the anchor point oj. The relationships
assumed to be spatially defined relative to this object, so that the predicted center of the query object is ô; oy Once trained we can produce samples from the MDN by evaluating it for the current observations and then applying standard GMM sampling using the output parameters.
As with the relation predictor Pp- the pose prior uses the object encoder to get lower-dimensional representations h, and hy for each object as well as the scene encoder to predict hz-
Scene discriminator The relation predictor and prior on their own are not enough to find stable placement poses. As such, we define discriminator network f(x, Z) trained to predict if given configuration results in placement that is physically realistic w.r.t. the training data, i.e. it is stable.
This model uses slightly different architecture from the above models, and unlike them first center the scene on o;, the query point and potential new pose for object i. We look at the local region 0.5m classify whether partieular 5 would result in realistic
around with fixed radius placement pose. We use a single PointNet++ model to encode points from both objects together, given label indicating which points belong the query object (the object whose placement we are attempting to find). In practice, the sphere-query necessary to extract the local context around particular pose the same as that used PointNet++, so this operation can be performed quickly at inference time.
Transform operator We require one additional operator to plan on point clouds: the transformation operator T (Z, D; ① 5). We assume deterministic transition function and rigidly transform points associated with the query object according to the relative pose offset å constructing new scene Z'.
Model Training We jointly train the object and scene encoders e and ez, relationship predictor Po. and the pose prior œ. The relationships can be directly supervised from training data, given

---

knowledge of the ground truth predicates, and are trained with binary cross entropy loss. The prior is trained to predict the offset from the anchor point oj the observed pose of the query object, Og, in the training data; i.e. 5 = Oj e
To train the scene discriminator, placements, thus there is no negative data nor stability supervision available. Instead, we create negative data online by applying random å offsets to the pose of the query object i. We simply sample a random ô and apply our transform operator to create a new scene Z T(Z, xi @ 6). These 8 were sampled be between 2 and 15 centimeters in random direction. The resulting scenes are highly likely to not be physically realistic nor stable (e.g. object is floating or is in collision).
first note that all of training data consists of stable object
Algorithm Placement planning algorithm pscudocode.
1: function FiNDPLACEMENT(ohjer point cloud 2; object point cloud zj, scene Z, goal Pij)
2:
for E range(1, N) do
3:
8, =
4:
while length (.) < B do
Rejectiio Sampling
5:
if s = 0 then
6:
8 x (I F.. Z, Piz)
Sample initial poses from learned prior
7:
else
8:
8 NN'',,,)) E)
Sample subsequent poses from surrogate distribution
9:
Z' + T(,,, +6)
>> Shift object point clouds by b Determine if pose is realistic >> Classify if goal predicates are true
realistic + f(; @ 2") > cf
11:
goal + Po(2; 1 0, Pipl > Ep
in in_view [I,, I 48)
> Ensure will be visible
13:
if realisticc and goal and in view then 808
14:
b Add new 5 to batch of samples
15:
Compute cost c(6) for each 8 5 Sort 8g and take top Nelite Fit surrogate distribution parameters (n',
17:
18:
return lowest-cost seen so far, or 0
2.2
Manipulation Planning with Relationship Models
We now describe how we solve the optimization problem for Equation using the components defined above. Alg. I describes how we can place object so to satisfy particular relationship. We take as given a set desired relationships phg in scene Z, and target objects and j. where the query object that will be moving and is the anchor object that will be kept stationary.
Initially, we perform rejection sampling draw batch of B candidate pose offsets from our MDN prior π (Z1, Fj Z, p.ú)) (line 6) keeping only those that satisfy the full set of constraints (lines 10-14). We sample until either a time budget has been reached or B samples have been successfully drawn.
For each sample we compute a new query point cloud using the transform function Z' T(Z, x; @ d) (line 9) and thus satisfy the con- straint in Eq. by construction. We then eval- uate the remaining constraints in Eq. 3-5 (lines 10-12) accepting only feasible samples (lines 13-14). Next, as per the cross-entropy method, evaluate the cost of the valid samples (line 15) and fit surrogate distribution to the best scoring Nelite samples with mean yí and vari- ance E' (lines 16-17). At each subsequent step, we draw samples ~ (N'''E'' from our current surrogate distribution, compute the scores again, and sample,, until we have per- formed N sampling iterations.
End
Figure 3: Depiction of the planning sequence. Initial samples drawn from MDN prior ò T(Z, Final poses satisfy both realism predicate con- straints finding a reasonable placement pose.
5

---

Fig. 3 gives an example for how the planning algorithm works in practice to achieve the goal that the yellow object be aligned to the right of the cyan object. In both Fig. 2 and Fig. 3, yellow objects are predicted positions of the query object; eyan represents the anchor object, and red points represent downsampled scene geometry. This an accurate depiction of the inputs into our model. The final frame of Fig. 3 has been rotated to show the precise alignment with the table surface.
2.3 Dataset Creation
We generated a large-scale dataset in simulation of RGB-D images with associated segmentation masks and relational predicates. We provide binary labels between all visible pairs of objects for all predicates listed in Table 2. Each scene consists of 3 to 7 random Shapenet [34] objects in stable configurations on various surfaces, including in stacks. We include mugs, bowls, and bottles, as well as boxes and cylinders of random sizes. Examples of generated scenes are shown in the supplemental material. Objects were placed in random configurations, and we ran physics forward to find stable arrangements. We then rendered images both with and without each object. In order to train the rotation model, we render each object on its own and apply a random rotation. Directional predicates (left of, right of. etc.) are computed based on bounding box overlap: others are computed based on distance between meshes or ground-trute position and orientation. See the supplementary materials for further details.
3 Experiments
We first performed set of simulation experiments on scenes that resembled our real world objects examining different versions of our model. First, we show an ablation test of our method on held- out YCB objects [35], with a set of known grasps from Eppner et al. [36], in order to show that our algorithm with discriminator is better able find stable positions for objects and scenes that did not appear in our training data. Second, we show a break-down of the predicate results, showing that our learned model comparable or better at capturing wide range of difficult relationships even in partially-oclluded scenes.
We sampled 100 random scenes in the kitchen environment from Fig. with objects either posi- tioned on top of the counter or in the top drawer. These objects did not appear in the training set. Table shows a breakdown of several different versions of the planner after running experiments on 100 random scenes, each with random predicate goal chosen from (in front, behind, left, right). Due to the random placements of the objects, not all scenes are feasible, and in many cases the goal pose would be occluded or off of the table to the front, resulting in challenging planning problems.
Variant
Found Predicates
Found Realistic
Successful
Stable Pose
Full model, A = 100
93
87
84
71
Full model, \ = 1
94
81
78
67
No Discriminator (Ay = 0)
100
3
3
25
Mean only
96
27
27
39
MDN Prior
99
6
6
42
Table 1: Comparison between different versions of the planner, when tested on 100 random multi-object scenes a kitchen environment. "Stable pose" is when the object center moved less than 5 cm after placement.
The different baselines look at the effects of the discriminator model, which determines whether or not a scene is realistic and whether not an object can be placed at particular pose. For example, the "no discriminator" case does not use the discriminator at all, and is very good at finding poses matching the predicate goal but not finding stable poses. We also vary the weight of the discriminator Ay in several examples. For these experiments we use a batch size, B, of 100.
We ran two experiments that do not use the discriminator in our "full" planning approach. Mean only draws samples only from the mean of the MDN prior π. This looks at what performance is like with a learned policy. MDN Prior uses the learned mixture density function as a cost function in place of using the discriminator, since presumably this might capture much of the same information. We can see that it actually does fairly good job at matching the predicates, but is not very discriminative when it comes to finding stable poses for placement. Both of these perform notably worse at finding stable poses in our test environments.

---

Left of
Right of
In Front
Behind
Above
Below
Near
Touching
Centered
Learned 0.92
0,93
0.74
0.65
0.90
0.92
0.88
0.90
0.70
Baseline 0.95
0.95
0.67
0.87
0.91
0.94
0.90
0.31
0.01
%True 13.9%
14.0%
4.6%
5.0%
4.3%
4.3%
29.0%
12.7%
5.6%
&False 86.1%
86.0%
95.4%
95.0%
95.7%
95.7%
71.0%
96.3%
84.4%
Table F1-score of the predicate predictor Pp held-out randomly-generatel simulated scenes. Some predicates in our scenes can be very difficult due to clutter and occlusions, but our learned models are either on par with or better than most all baselines. Bottom two rows show prevalence in the evaluation data set.
3.1 Scene Discriminator Performance
Here, we examine the performance of our scene discriminator. To do this, we compare placement poses sampled from the MDN prior distribution in randomly-generann ment scores after optimization. We place the object the new pose and then run 500 simulation steps to allow the object to settle into its final pose. We then compare the discriminator's confidence score with how much the object moved. For these experiments we generated 100 random scenes, each with random predicate goal so as not to bias it to particular subset of the problem space. We ignore scenes if no feasible pose was found according to the discriminator.
kitchen scenarios to place-
We found that in 95% of these scenarios, the discriminator was able to find a stable pose to place particular object, and in 90% of all scenarios the planner was also able match the specified predicates. This shows that not only can we find realistic positions, but that the discriminator does not preclude achieving specified goals. Note the higher perceived success rates than in the planner comparison in Table 1: this is because we only test scenes where the planner was initially confident that it could find a solution.
Realism vs. Observed Stability
"BELOW"
"IN FRONT"
"RIGHT"
"ABOVE"
EXAMPLE FAILURES
2:2 003
Figure 4: Sim experiment results. Left: correlation between predicted realism (y axis) and distance moved after placement axis). The blue dots show randomly sampled poses, while the red stars indicate poses after rumning the placement planner. More realistic poses are much more stable than less realistic ones, although in some cases unrealistic poses do not move much either. Right: selected successful and unsuccessful simulation results. Images show point clouds from the robot's point of view, after planning.
Fig. (left) shows the relationship between the distance the object moved once placed and the output of the scene discriminator model, after rejecting small number of outliers (distance > 0.5 meters). When we compute Pearson's 's correlation coefficient r. we see that for a single random sample 8, r. -0.147 with p. 0.161, and for the final predicted placement P, rp -0.278 with pr = 0.006. Individual initial samples s are shown blue and optimized placements in red. These results indicate that our scene discriminator learns a metric which correlates with stability, even in challenging, cluttered scenes. Failures of the discriminator are driven by oddly perceived object environment geometry. When parts of the object geometry are missing, the model is prone to making mistakes, such as placing an object so that intersects with another object, or placing an object so that it is not properly supported, as shown in Fig. 4 (right).
3.2 Predicate Performance
Table 2 shows the Fl score by predicate held-out portion of the dataset containing 3000 exam- ples. Our model correctly predicts every predicate 69.4% of scenes, vs. 58.5% for the baseline. In addition, can learn predicates for which we have baseline, such as aligned, on which we have
7

---

true positive rate of 83% (f1 = 0.43). Generally, our model classifies different relationships well, matching or exceeding the baseline in every case but one. In particular, our learned model has con- sistently good sensitivity, meaning that we can find valid positions for the object when attempting to plan. Finally, the rule-based approach requires implementing and hand-tuning a range of differ- ent predicates; by learning from data, we could in principle scale to a larger number of symbolic relationships. Our results show that even with very unbalanced, "natural" data, we can learn useful classifiers for range of relationships.
First, we compute bounding boxes and means from each object point cloud. For the centered predicate, we compute zy distance on the table and use larger threshold that appears in our dataset (2 mm instead of mm). For touching, we compute minimum distance between point clouds and threshold it with 2.5mm. For near, the threshold is a em distance. For di- rectional predicates, we apply the exact same rules used in data generation to the computed bounding boxes, as seen in the supplementary materials.
Baseline implementation:
3.3 Real World Experiments
Finally, we deployed our system on Franka Panda robot with an arm-mounted RGB-D camera. We used objects from the YCB object set [35] augmented with toy kitchen objects. We also used various cardboard boxes to force the robot to adapt to changes in scene geometry. We generate grasps using [29] and use RRT-Connect [37] for motion planning. We used unseen object instance segmentation from [12] to determine segmentation masks for different objects, and use a prompt to choose which object to move when performing experiments. We compute standoff poses for each object 10cm above the predicted goal position, and release 2cm above the predicted pose order to ensure that we do not press into the table. Figure/ I shows example pairs of before-after images.
To quantify our results in the real world, we conducted a of experiments with eight different objects. We report results in Table 3. Our placement planner was highly successful at finding ma- nipulation plans with range of different objects and predicates, including both grasps and place- ments for the various held-out objects. Crucially, our method is able to find multiple valid solutions for each scene; some examples appear in the supplementary materials. We saw a high rate of grasp execution failures on the real world, which could easily be improved in future work by re-grasping,
Table Only Grasp
Table with Large Box Grasp
Success Rate
Plan
Placement
Plan
Placement
Sauce Bottle
3
2
2
2
1
Cookic Box
3
3
3
0
Red Mug
3
I 1
Juice Carton
Macaroni Box
3
2
2
0
Parmesan Can
3
0
0
Mustard Bottle
3
3
2
Large Pitcher
2
0
Overall (%)
95%
75%
100%
75%
56%
60%
Table 3: Generalization experiments with different objects. All numbers out of three trials. Placement successes conditioned on successful grasps. We see that the largest cause of failures was grasping issues. Planning failed a few situations with challenging objects that were either very large (YCB pitcher) or very small and hard to grasp (Parmesan can). Placement fails when an object falls off of the box after the planner attempts placement.
4 Conclusions and Future Work
We demonstrated the ability for a robot to learn to accurately infer inter-object relations from point clouds of real-world, unstructured environments, which can be used perform manipulation plan- ning for previously unseen objects. Our results show that model trained purely in simulation, effectively predicts relations on real-world point clouds of objects not seen during training. Further- more, our incorporation of a scene-realism discriminator significantly improves performance over predicate goal predictor alone. the future, we will add semantic understanding about which objects the robot is interacting with using learned object class and attribute classifiers. We will also expand our method to handle multi-object relations and explore long-horizon manipulation tasks by formally extending our method into a task and motion planner.
8

---

Acknowledgements
Chris Xie was funded by NSF NRI grant IIS-2024057.
References
[1] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Péred Integrated task and motion planning. Annual Review of Control, Roboties, and Autonomous Systems, 2021.
[2] S. Srinivasa, A. M. Johnson, G. Lee, M. Koval, S. Choudhury, J. King, Dellin, M. Harding, D. Butterworth, P. Velagapudi, and A. Thackston. A system for multi-step mobile manipula- tion. Int. Symp. on Experimental Robotics, 2016.
[3] D. Batra, A. X. Chang, S. Chernova, A. J. Davison, J. Deng, V. Koltun, S. Levine, J. Ma- lik, Mordatch, R. Mottaghi, et al. Rearrangement A challenge for embodied AL. arXiv:2011.01975 2020.
[4] D. Xu, R. Martín-Martin, D.-A. Huang, Y. Zhu, S. Savarese, and L. Fei-Fei. Regression planning networks. In Neural Info. Proc. Sys., 2019.
[5] A. H. Qureshi, A. Mousavian, C. Paxton, M. C. Yip, and D. Fox. NeRP: Neural Rearrangement Planning for Unknown Objects. In Robotics: Science & Systems, 2021.
[6] D.-. Huang, D. Xu, Y. Zhu, A. Garg, S. Savarese, L. Fei-Fei. and J. C. Niebles. Continuous relaxation symbolic planner for one-shot imitation learning. Intl. Conf. on Intelligent Robots and Systems, 2019.
[7] C. Paxton, Barnoy, Katyal, R. Arora, and D. Hager. Visual robot task planning. In Intl. Conf. on Robotics and Automation, 2019.
[8] K. Kase, C. Paxton. H. Mazhar, T. Ogata, and D. Fox. Transfcrable task exccution from pixels through deep planning domain learning. Intl. Conf. Robotics and Automation, 2020.
[9] Y. Zhu, J. Tremblay, S. Birchfield, and Y. Zhu. Hierarchical Planning for Long-Horien Ma- nipulation with Geometric and Symbolic Scene Graphs. In Intl. Conf. Robotics and Au- tomation, 2021.
[10] D. Huang, S. Nair, D. Xu, Y. Zhu, A. Garg, L. Fei-Fei, S. Savarese, and J. C. Niebles. Neural task graphs: Generalizing to unseen tasks from single video demonstration. In Conference on Computer Vision and Pattern Recognition, 2019.
[11] D. Xu, A. Mandlekar, R. Martín-Martín, Y. Zhu, S. Savarese, and L. Fei-Fei. Deep affor- dance foresight: Planning through what can be done the future. Intl, Conf. on Robotics and Automation, 2021.
[12] Y. Xiang, C. Xie, A. Mousavian, and D. Fox. Learning rgb-d feature embeddings for unseen object instance segmentation. In Conf. Robot Learning, 2020.
[13] M. Forbes, R. P. Rao, L. Zettlemoyer, and M. Cakmak. Robot programming by demonstration with situated spatial language understanding. In Intl. Conf. on Robotics and Automation, 2015.
[14] B. Rosman and S. Ramamoorthy. Learning spatial relationships between objects. Inil. Journal of Robotics Research, 2011.
[15] M. Clement, C. Kurtz, and L. Wendling. Learning spatial relations and shapes for structural object description and scene recognition. Pattern Recognition. 2018.
[16] O. Mees, A. Emek, J. Vertens, and W. Burgard. Learning object placements for relational instructions by hallucinating scene representations. Intl. Conf. on Robotics and Automation, 2020.

---

[17] D. M. Bear, C. Fan, D. Mrowca, Y. Li, S. Alter, A. Nayebi, J. Schwartz, L. Fei-Fei, J. Wu, J. B. Tenenbaum, al. Learning physical graph representations from visual scenes. Neural Info. Proc. Sys., 2020.
[18] S. Panda, A. A. Hafez, and Jawahar. Learning support order for manipulation clutter. In Intl. Conf. on Intelligent Robots and Systems, 2013.
[19] H. Zhang, X. Lan, S. Bai, L. Wan, C. Yang, and N. Zheng. Multi-task Convolutional Neu- ral Network for Autonomous Robotic Grasping in Object Stacking Scenes. In Intl. Conf. on Intelligent Robots and Systems, 2019.
[20] R. Paul, J. Arkin, N. Roy, and T. M Howard. Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators. In Robotics: Science & Systems, 2016.
[21] S. G. Venkatesh, A. Biswas, R. Upadrashta, V. Srinivasan, P. Talukdar, and B. Amrutur, Spatial reasoning from natural language instructions for robot manipulation. In Inil. Conf. on Robolics and Automation, 2021.
[22] M. Sharma and O, Kroemer. Relational learning for skill preconditions. Conf. on Robot Learning, 2020.
[23] M. Wilson and T. Hermans. Learning to Manipulate Object Collections Using Grounded State Representations. In Conf. on Robot Learning, 2019.
[24] Y. Jiang, M. Lim, C. Zheng, and A. Saxena. Learning to place new objects in scene. Inil. Journal of Robotics Research, 2012.
[25] J. Baumgartl, T. Werner, P. Kaminsky, and D. Henrich. A fast, gpu-based geometrical place- ment planner for unknown modelr-dd objects and placement areas. In Intl. Conf. on Robotics and Automation, 2014.
[26] F. Furrer, M. Wermelinger, H. Yoshida, F. Gramazio, M. Kohler, R. Siegwart, and M. Hutter. Autonomous robotic stone stacking with online next best object target pose planning. In Intl. Conf. on Robotics and Automation, 2017.
[27] R. Newbury, K. He, A. Cosgun, and T. Drummond. Learning place objects flat surfaces in upright orientations. IEEE Robotics and Automation Letters, 2021.
[28] Q. Lu, M. Van der Merwe, B. Sundaralingam, and T. Hermans. Multi-fingered grasp planning via inference in deep neural networks, IEEE Robotics & Automaaton Magazine, 2020.
[29] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Grasp Generation in Cluttered Scenes. In Intl. Conf. on Robotics and Automation, 2021.
Efficient 6-DoF
[30] M. Kobilarov. Cross-entropy motion planning. Intl. Journal of Robotics Research, 31(7), 2012.
[31] C. Paxton, F. Jonathan, M. Kobilarov, and G. D. Hager. Do what want, not what did: Imitation of skills by planning sequences of actions. In Intl. Conf. on Intelligent Robots and Systems, 2016.
[32] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In Neural Info. Proc. Sys., 2017.
[33] C. M. Bishop. Mixture density networks. Tech Report, Aston University, 1994.
[34] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rict 3D Model Reposi- tory. arXiv: 152.00012. 2015.
[35] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In Intl. Conf. on Advanced Robotics, 2015.
[36] C. Eppner, A. Mousavian, and D. Fox. A billion ways to grasp: An evaluation of grasp sampling schemes on a dense, physics bhssecs-based grasp data set. arXiv:19/2.05604 2019,
[37] J. Kuffner and S. M. Valle. Rrt-connect: efficient approach to single-query path plan- ning. In Intl. Conf. on Robotics and Automation, 2000.
10


Reviews:
[{"review_id": "RncU7b278B-", "reviewer": "Reviewer_f3r9", "summary": "The paper proposes a novel approach for predicting stable placements of objects in a rearrangement task. This goal of this task is to rearrange a set of objects given high-level predicates describing their target location such as \"above\", \"near\" etc. The paper decomposes this task into multiple subcomponents: (1) Object and scene encoders which project the point clouds of a pair of objects and the scene into a latent space, (2) a relation predictor that predicts a set of fixed spatial binary predicates from the latent space representations the objects and scene, (3) a pose prior used to sample new object locations given a spatial predicate, (4) a scene discriminator used to reject invalid placement proposals, (5) a placement planner to determine a target placement location.\n\nThe paper evaluates the approach in a variety of simulation and real-world experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe paper addresses a challenging an important problem and proposes a complex and technically sound approach. It provides a large set of both simulated and real-world experiments, and shows the general validity of the approach. The presentation is mostly clear and easy to follow. I like the decomposition of the approach and the inclusion of a scene discriminator as a \"critic\".\n\n## Weaknesses\n\n1. Presentation and experimental evaluation: The experimental sections contain various studies, and I think it is a good idea to re-order the sections as done in the revision. I now understand what the key result is and why it shows that the proposed method works better. However, the paper would still profit from a clearer guidance for the reader as to what the experimental hypotheses are, how they matter for the final performance, and whether the experiments confirm the hypotheses. \n\n2. Model-free vs. model-based. As discussed in the rebuttal, the dataset generation (Section 2.3) and the model-based baseline are large equivalent with the main difference being that one operates on meshes and one of partial points clouds, and that the model-based baseline requires access to CAD models. The latter difference was not obvious to me, and should be pointed out explicitly in the paper.\n\n3. Justification on decisions for problem formulation. In their rebuttal, the authors argued that multiple cameras are difficult in non-controlled scenarios (I agree) and that collecting multiple viewpoints is too difficult / costly (I don't fully agree). I feel the paper makes some arbitrary choices as to what it expects as given and how it frames the problem - e.g. instance segmentation capabilities are considered as a given, but a dense fusion capability for acquiring and merging multiple viewpoints is not. The paper would profit from a better justification as to why these choices were made.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a complex yet mostly sound approach to the complex rearrangement task. The paper has some shortcomings wrt to clarity and the experimental evaluation, in particular in terms of its presentation.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "### Clarity & Presentation\n\n- Introduction: I couldn't really understand from the introduction what the problem the paper wants to solve and how it wants to solve it until reading section 2.\nImportantly, the introduction does not explain what the rearrangement task actually *is*, and the argument in L24-29 does not really motivate the task of rearrangement. I would recommend to first explain what the rearrangement task is, why it is challenging, and how this paper decomposes this task. Then it should be explained what alternatives exist (incl prior work) for each subcomponent, and what the choice + contribution of each component of this paper is. I feel that all of this information is somewhere hidden in the introduction, but it needs to be reordered/rewritten in order to guide the reader more gently through the complex matter.\n\n- Figure 2 & 3: I cannot really understand the depictions of the MDN. What do the different colors mean? What do we see here? These figures should be explained in the paper.\n\n- Latent object & scene encoders. First of all, I think it should be stated more clearly what the purpose of these is. If I understand correctly, this component enables the approach to generalize over different object geometries. Second, I'm wondering what role the hard-coded radius delta plays in the generality of the approach. \n\n### Experimental evaluation\n\nTable 1 reports compares the learned and the model-based baseline in terms of specificity and sensitivity, but it is actually hard to tell which one performs better. Eyeballing the numbers, it doesn't seem to me that one can be regarded the clear winner, depending on whether we put more emphasis on specificity or sensitivity. Which one should we give more weight?\n(Also, what does it mean for the model-based baseline for \"centered\" to have 0% sensitivity and 100% specificity?)\n\nI would suggest to use a single metric (e.g. F-Score) to compare the baselines. Furthermore, I would suggest to use bold face for the best performing approach in each row.\n\n### Dataset generation & model-based relationship predictor\n\nIt sounds to me that the dataset generation (Section 2.3) and the model-based baseline are large equivalent with the difference that one operates on meshes and the other on partial points clouds. Is this assessment correct? Or does object geometry play a bigger role? I would like to see in the paper a more explicit explanation of the differences.\n\nIf that assessment is correct, it seems that the only problem of the model-based baseline is the partial visibility. I would like to see a more elaborate discussion in section 3 if that is the case, or why else the model-based baseline performs (allegedly, see previous point) worse.\n\nFinally, if the partial visibility is a problem, why not add 1-2 additional RGB-D sensors? It should be justified why this is not a solution to the problem, as it would significantly simplify the machinery proposed here to infer spatial relationships.\n\n### Minor issues\n\n- Real world experiments. The paper makes the assumption that object points clouds are fully segmented, but no details apart from a citation of [12] are mentioned in section 3.3. The section mentions that grasping is an issue, but can failures also be attributed to the segmentation?\n- L99 \"optima\" is plura, the singular form is of \"optimum\"\n"}}, {"review_id": "G8Ok4tDbsfc", "reviewer": "Reviewer_Tchz", "summary": "This paper presents a novel integrated approach combining visual perception, logical reasoning, and motion planning of a robot arm for pick-and-place tasks. \nLearned through segmented RGB-D data from simulated scenarios, semantic spatial relations between objects are inferred. \nA novel discriminator function is also learned to predict the feasibility of placement pose of an object given a semantic goal defined as a relative spatial relation between the query and a secondary anchor object.\nThe output of this combined visual and semantic module is fed into an of-the-shelf sampling-based motion planner to realize the pick-and-place task by the robot. \nThis architecture is also shown to be transferable to a physical robot executing such plans in the real-world.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this work is the novel combination of the learned (spatial) relational predictor and the physical validity discriminator. The overall system architecture is also well-engineered. \n\nExperimental analysis is detailed and supports the novelty and usefulness claims. The analysis and justification of failure cases are also clear. The real-world transfer of learned components is valuable and encouraging.\n\nIn terms of improvement points, I have a few suggestions:\nThe emphasis on the applicability of the proposed approach for 'unseen' objects is not justified. Because i) it's unclear which type of objects were in the training set vs the leftout 'unseen' ones and how much they resemble/differ from each other w.r.t. their geometries, and ii) the framework relies on segmented point cloud data, 'objectness' seems to become irrelevant or less important. \n\nDecoupling of goal pose prediction/planning and motion control/execution is a critical (design) feature of this work. I can see its benefits, however, many robotic manipulation tasks require sequential decisions that are likely to depend on each other not only in terms of logical validity but also for manipulation planning feasibility. For example, how (with which pose) the robot picks up a tool to use affects the feasible actions for the follow-up steps. Since the authors also suggest investigating long-horizon tasks, the paper can benefit from such a discussion on how the proposed approach can be applied on those scenarios. \n\nIt's also unclear how this approach can be extended for multi-object relational predictions and planning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a two-step process for the problem of rearranging objects in an environment by robots.\nThis strategy comprises learning to predict spatial semantic relations between objects and learning to infer physical validity of placement poses using RGB-D sensory data. \nEven though individual components rely on prior works, their combination provides a novel framework for simple pick-and-place tasks given spatial logical goals.\nThe experimental analysis demonstrates promising initial steps for the applicability of this approach for such problems.\nIn general, the paper is well-written, and with some revisions and clarifications it can be improved further.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Some of my suggestions are above (Strengths & Weaknesses). Here, I provide some further improvement points.\nOverall, the presentation is good. However, the method section encapsulates both prior work that this paper relies on and the original contribution all together. Either a separation or clear emphasis on what is based on prior work vs new would improve the paper.\n\nThe role of the scene encoder is not well motivated/justified and thus might be clarified better.\n\nSupp. material:\n\nVideo: Timings need improvement. There is some flickering.\n\nAppendix: There are some typos, unfinished sentences.\n"}}, {"review_id": "4TGIQIU3dzt", "reviewer": "Reviewer_54xG", "summary": "The paper proposed an approach to manipulate unknown object with vague instructions. The proposed method learns the topology relationship between an anchor object and the unknown object (segmented though). ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Learning topology and manipulated with vague instructions could improve human robot interactions significantly, and such technique can enable many robotics applications.\nHowever the vague instructions could add more uncertainty and I suspect the affected range surrounding the anchor object might be an sensitive parameters and have strong effects on the performance of the proposed approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well organized and easy to follow, and the proposed approach was verified with real world experiments.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "The 'unknown objects' in the title could be clarified a bit. e.g. the main focus is to be able to manipulated 'novel' object but still rely on segmentation performance.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new approach for predicting stable placements of objects in a rearrangement task, based on ambiguous instructions that specify the spatial relations between the placed objects and the surrounding "anchor" objects. The proposed method decomposes  this task into multiple subcomponents: (1) object and scene encoders, (2) relation predictor, (3) a pose prior used to sample new object locations given a spatial predicate, (4) a scene discriminator used to reject invalid placement proposals, (5) a placement planner to determine a target placement location. 

While the paper addresses a challenging problem and proposes a complex and technically sound approach, there were some issues in the paper in its first version, as reported by the reviewers. Mainly, the paper was difficult to understand. The results reported in the experimental evaluation also seemed to be mixed and did not clearly indicate which method works better. 

In their rebuttal, the authors updated the paper and resolved most of the mentioned issues, especially in terms of presentation. The authors also added new results and metrics, such as the F1 score, that clearly indicate the advantage of the proposed solution.
Decision: Accept (Poster)