Learning to Regrasp by Learning to Place
Authors: Shuo Cheng
Kaichun Mo
Lin Shao
Venue: CoRL2021 Poster (2021)
Keywords: Regrasping
Deep Learning
Robotic Manipulation

Abstract: In this paper, we explore whether a robot can learn to regrasp a diverse set of objects to achieve various desired grasp poses. Regrasping is needed whenever a robot's current grasp pose fails to perform desired manipulation tasks. Endowing robots with such an ability has applications in many domains such as manufacturing or domestic services. Yet, it is a challenging task due to the large diversity of geometry in everyday objects and the high dimensionality of the state and action space. In this paper, we propose a system for robots to take partial point clouds of an object and the supporting environment as inputs and output a sequence of pick-and-place operations to transform an initial object grasp pose to the desired object grasp poses. The key technique includes a neural stable placement predictor and a regrasp graph based solution through leveraging and changing the surrounding environment. We introduce a new and challenging synthetic dataset for learning and evaluating the proposed approach. We demonstrate the effectiveness of our proposed system with both simulator and real-world experiments. More videos and visualization examples are available on our project https://sites.google.com/view/regrasp.

Learning to Regrasp by Learning to Place
Shuo Cheng', Kaichun Mo², Lin Shao?
University of California, San Diego - schenggeng ucsd. edu 2 Stanford University - {kai chunm, sins2)0stann ord. edu
Abstract: In this paper, we explore whether robot can learn to regrasp di- verse set of objects to achieve various desired grasp poses. Regrasping is needed whenever a robot's current grasp pose fails to perform desired manipulation tasks. Endowing robots with such an ability has applications in many domains such as manufacturing or domestic services. Yet, is a challenging task due to the large diversity of geometry in everyday objects and the high dimensionality of the state and action space. In this paper, we propose a system for robots to take partial point clouds of an object and the supporting environment inputs and output a sequence of pick-and-place operations to transform an initial object grasp pose the desired object grasp poses. The key technique includes a neural stable placement predictor and a regrasp graph based solution through leveraging and changing the surrounding environment. We introduce thetic dataset for learning and evaluating the proposed approach. We demonstrate the effectiveness of our proposed system with both simulator and real-world ex- periments. More videos and visualization examples are available on our project webpage
and challenging syn-
Keywords: Regrasping, Object Placement, Robotic Manipulation
Introduction
"Regrasping must be performed whenever robot's grasp of an object is not compatible with the task it must perform." [1]
Through regrasping, object can be picked up initially, placed down
intermediate stable state possibly supported by a second object or the environment geometry, and then be grasped in a target pose and position. Regrasping is common daily task and plays crucial role in connecting vari- ous manipulation operations to solve complex long-horizon tasks. For example, when assembling parts to IKEA chair, we may need to pick-and-place these parts several times to find appropriate orientations before mating and putting them together. When preparing the food, we may need to pick-and-place the knife and spatula several times find the desired grasp poses for tool usages. The ability of a robot to regrasp a wide variety of objects is of tremendous importance for many application domains such as manufacturing or domestic services. However, the large diversity of geometry in everyday objects and the high dimensionality of the state and action space make this challenging manipulation task. In this paper, enable a robot construct sequence of pick-and- place operations to achieve various desired grasp poses.
Picking objects, or grasping, has attracted great attention in roboties [2, 3, 4, 5, 6, 7]. In contrast, grasping, which is generating the sequence of pick-and and-place operations, has received considerably (GP) search for regrasp sequences.
less attention. Early work [1, 8] builds a Grasp-Placemen It first fills valid grasp and placement pairs in the GP table and then searches the table to find a sequence of pick-and-place motion. There are many works following this direction [9, 10, 11]. It usually requires a full observability of the environment and takes a significantly long time before finding a solution given the high-diimnsionnlity of the searching space. Jiang et al. [12] use Support Vector Machines with hand-designed features to score the placement suitability of candidate poses and they focus on the object placement. this paper, we propose a novel stable object pose pre- diction model with deep-learned features extracted directly from the raw point clouds and consider both picking and placing operations for regrasping tasks.
Taking as inputs partial observations of the object to regrasp and the surrounding environment, our system predicts pick-and-plac, sequence reach the desired object grasp poses the output. The
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Figure I: Regrasping must be performed an initial grasp pose (top-left) is not suitable for down- stream task. Taking imputs the partial observation of the object (the green spoon) and the sur- rounding environment (the ground and a blue mug), as well as the desired target grasp pose that is more suitable for the downstream task (bottom-left), ment and achieve the regrasping task through the following steps: a) Place the spoon because the initial grasp is not reachable to the target grasp pose; Grasp the mug, c) Place the mug struct a stable supporting environment, d) Grasp the spoon, e) Place the spoon with stable pose that feasible for regrasping. f) Grasp the spoon by up and down for matching the target grasp pose.
system helps robot to adjust the environ-
system contains learned object placement module that is able efficiently generate diverse set of valid object poses for placement, and regrasp framework that autonomously creates a regrasp graph to search for desired grasp poses by leveraging and changing the surrounding environment.
Our primary contributions are: proposing a novel manipulation system that enables robots re- grasp a diverse set of objects through actively leveraging and changing the surrounding environment, 2) introducing a learning-ban
model to predict stable object placement poses; 3) generating an an- notated dataset for object regrasping tasks and conducting extensive experiments to demonstrate the effectiveness of our proposed approach.
2 Related Work
We review literature related to the two key components in our approach predicting object placement and planning regrasp graph, and describe how we are different from previous works.
2.1 Object Placement
Tick-and-plan one of the most common robotic manipulation tasks. Picking objects, or grasping, has attracted great attention in robotics. For a broader review of the field on data-driven grasp synthesis, we refer [13, 7]. contrast, object placement, which is the process of deciding where and how to place an object, has received considerably less attention. Harada et al. [14] presented an object placement planner find object placement poses by matching planar surface patches on the object with planar surface patches in the environment. Haustein et al. [15] introduced motion planning algorithm for robots to place grasped object in cluttered environment. Jiang et al. [12] employed Support Vector Machines with hand-designed features to score the placement suitability of candidate poses. Unlike these works either requiring full observability or relying on hand-crafted feature engineering, we propose an end-to-end approach with deep-learned features extracted directly from the raw point clouds.
2.2 Regrasp for Object Reorientation
Early works [1, 8] build Grasp-Placenen (GP) to search for regrasp sequences, by solving the in- verse kinematics, checking all collisions to remove invalid plasp-placement pairs, filling them in the table, and finally searching within the table to find a sequence of andp-and motion. There are many works following this direction. Rohrdanz and Wahl [9] improved the efficiency of regrasp planning using an evaluated mreadth-fisstseene rated grasp, and placement quantities. Stoeter al. [10] replaced the GP table with space of compatible grasp-placement-geee grasp triplet and searches

---

this space to find a sequence of lock-and-llas motion. Regrasp is also studied in the context of task and motion planning (TAMP). Lozano-Pérez and Kaclbling [16] presented a framework where a symbolic planner plans a sequence of high-level sub-tasks and constraint satisfaction problem solver plans low-level operations. A regrasp plan composed of symbolic pick-and-place se- quence and a set of geometrically feasible placements, grasps, and paths, which requires precise full observability of the entire system.
Wan et al. [11] presented a regrasp planning component for object reorientation. Given the initial and goal pose of an object, the regrasp planning component finds sequence of robot postures and grasp configurations that reorient the object from the initial pose to the goal pose. Raessa et al. [17] presented a hierarchical motion planner for planning the manipulation motion to re-pose long and heavy objects. They developed a graph-basee based planning system that combines both regrasp and in-hand manipulation motion considering external support surfaces. Cao et al. [18] increased the reorientation capability of pick-and-place regrasp by adding a vertical pin on the working surface and using as the intermediate location for regrasping. The work most related to ours [19] where they developed regrasp planning algorithm considering intermediate stable states fixture such as box baskets. Unlike these planning based approaches, our learning-based approach do not require full observability and learn features directly from raw sensory data. Our approach also allows robots to actively change other objects in the environment to create extra supports. Besides regrasping, object reorientation can also be achieved through in-hand manipulation [20, 21, 22] dual-arm manipulation [23, 24].
3 Problem Definition
In this section, we formulate the object regrasping problem and introduce necessary concepts we use. Note that there may be various other ways to parameterize the same problem.
We consider a single-arm robot equipped with a two-fingered robotic hand. However, our proposed pipeline can be applied to many other settings, such as multi-arm robots or dexterous robotic grip- pers, to enhance their manipulation performances. We assume that the segmentation of all objects within the surrounding environments is provided and the objects are graspable for the robotic hand. Note that our approach could also be extended to movable but ungraspable objects in the surrounding environment leveraging non-peehen manipulation processes such pushing.
Notations. Let D = { denote the set of objects in the scene, including the target object Or supporting objects Oizt. and the ground Og- We use H. R. and C to denote the robotic hand, the robot, and the robot configuration respectively. Each object O; and the robotic hand H has its own local coordinate frame T and TH Their pairwise relative transformations are also known. Let T g
and T denote the 6D poses of the object O2 and the robotic hand H in the world frame To,
Grasping Operation. A grasp pose G = (pk, q d of an object O; is parameterized by two contact points on the object surface (pi. q and an approaching direction of the robotic hand dj All three vectors are described in the object local frame T'. A grasping operation Li depends on the grasp pose Gj. the object pose in the world frame Ty. the surrounding environment D - {O.). and the robot's configuration C. A grasping operation & is valid if the contact points are in force- closure, reachable by the robot, and occurring no collision in the process. A valid grasp Sj attaches the object O to the robotic hand H. and hence its 6D pose in the world frame can change as the robotic hand moves. We assume all objects, including the supporting objects Oitt,, can be grasped, since the robot may rearrange the environment in various ways to support the target object O, finally.
Placement Operation. When an object O is placed stably on a surrounding environment, we use its global world pose T to describe its placement pose. placement operation Pi of the object O, depends on the placement pose Th the surrounding environment D [O.]. and the robot's configuration C. The placement operation Si is valid if 1) the placement pose is reachable by the robot, 2) there is no collision, and 3) the object remains stable after the robotic hand releases the object. A valid placement Pt detaches the object O from the robotic hand. Its 6D pose in the world frame 7 remains when the robot moves away.
The Regrasping Problem. The problem of regrasping object O is defined as finding sequence of valid grasping operations & and placement operations S over all objects { to reach target grasp pose G for the target object O,
3

---

Pick and Placo the Mug?
Initial Scene
Pick Spoon?
Target Grasp Pose
Found Target Grasp Pose
Figure 2: Illustration of our overview pipeline that enables robot to achieve the regrasping task through adjusting the environment step by step. Given the partial observation of all objects and the surrounding environment, our system leverages the learned neural network propose diverse distribution of object poses for stable placement, and it utilizes pose classifier to remove unstable pose proposals. Our system gradually constructs search graph for finding valid sequence of object pick and place operations to allow the robot reach the final target grasp pose.
4 Technical Approach
this section, we introduce our regrasping pipeline, consisting of a plan searching algorithm over regrasp graph (Fig. 2) and deep-learned model for stable pose placement prediction (Fig. 3).
4.1 Regrasp Graph Construction and Scarching
regrasping problem solves sequence of valid grasping and placement S operations to reach target grasp pose G. of the object O. We also allow robots to change other objects' poses (T/+t) by applying several intermediate grasping-and and-placing operations, since changing the poses of other objects rearranges the environment and may provide extra supports that increase the number of valid grasping operations for the target object O,. Below, we describe in details how to construct a regrasp graph and searching algorithm over figure out plan.
Graph Construction. Our system creates and leverages a regrasp graph to solve the regrasping problem. Each node A in the regrasp graph includes stable poses (T₂) for all objects (O,). An edge Ene connects two nodes N₂ and N there one and only one object O, whose stable pose is changed between the two nodes, and there exists valid pair of grasping operation & and placement operation S that transform the object's stable pose. The edge indicates that the robot can first grasp the object O from its stable pose under one node and then place it at the stable pose the other node.
Initially, all objects {O,} are stably placed in the scene. All objects' initial poses define the first node in the regrasp graph. For each object Oj. we pose prediction model (Sec. 4.2) to estimate the object's alternative stable poses under the current surrounding environment. Each alternative stable pose of the object Oj, along with other objects' stable poses, leads to a new node in the regrasp graph. Whenever a new node created, our system checks whether there are edges connecting it to the rest nodes and add these edges the graph.
deep-learned stable object placement
Graph Searching. Leveraging the constructed regrasp graph, the solution of a regrasp problem is a path from a starting node Ng target node Nf that contains a stable object placement pose T allowing valid grasping operation Jf to achieve the target grasp pose G for the target object O,, We use Sapphh-frst-searde algorithm to search for valid path.
4.2 Stable Object Poses Prediction
We train a neural network to predict stable object placement poses under diverse surrounding envi- ronments. Estimating stable placement poses for wide variety of objects with diverse and compli-

---

Stable Pose Proposal
Stable Pose Classification
PointNet++ encoder
MLP
Stable?
MLP
In Contner?
PointNet encoder
PointNct++ encoder
FOSR
MLP
Tym
PointNet*+ decoder
MLP
Contact Points?
Figure 3: Our proposed framework for predicting stable object placement poses. As shown in the left figure, first learn stable pose proposal network that takes inputs the partial point clouds of the object and the surrounding environment, and outputs a diverse set of rough pose estimations. We formulate diverse pose prediction outputs. On the right figure, we then show our stable pose classification and refinement network that learns to classify an object 6D pose leads to a stable placement. We then apply CEM [25] to perform local optimization for the pose refinement. We also leverage multi-task training by predicting the contact points between the object and environment
generative model and feed in different Gaussian noise vectors 2's query
find beneficial.
cated geometry is a challenging task. requires very accurate pose predictions, since slight error may easily result in an undesired object falling or penetration between the object and its surrounding environment. Hence, we design a two-stage pipeline (Fig. 3) that first predicts diverse rough poses and then refines to obtain more accurate results. We describe the detailed network designs below.
Stable Poses Proposal. Given an object O and static surrounding environment S D {O,), we first train generative model that learns to propose a diverse set of poses (Th) } for the object O be stably placed in the environment S.
Fig. (left) illustrates the inputs, outputs, and design overview for this network. In our imple- mentation, represent both the partial observations of the object X, Rmx3 and surrounding environment X E Rmmx3 captured by the robot's RGB-D camera, as 3D) point clouds consisting of 1, 024 points. We first employ two PointNet++ encoders [26] to extract global point cloud features f. E IR 128 and fs E IR 128
To propose multiple solutions to the stable poses of the object O, we sample random Gaussian noise E R as an additional input and use an Multilayer percep- tron (MLP) to extract a feature f- E R64, We concatenate fi, fs. and fz to produce a single joint feature ft.S.z and finally employ another MLP to decode a final 6-DoF pose F g. for the object O,- For brevity, we omit the subscripts g denoting the global world coordinate, S indicating the environment, and the superscript representing the shape O in the following paragraph. The 6-DoP pose composed of a 3D translation and 3D orientation. For the orientation part, we adopt the axis-angle representation, which has been shown to be effective for pose prediction task [27].
Sampling different Gaussian noises {aI we obtain a diverse set of pose predictions { Y Provided with a ground-trutt list of stable poses [T2, )i= generated by running several random interaction trials in physical simulation, we define a loss Lm to train our pose prediction outputs.
LM (
(Ty Wim) Σ min L(T.. Tzy) +
min L(T Tay)
(1)
where L measures the distance between 6-DoF poses. We implement L as the average La distance between the corresponding points of the object point clouds after applying the two poses.
Stable Pose Classification and Refinement. We train a model classify the stability of the sam- pled poses. For each predicted pose T., first concatenate the transformed object point cloud T., (X.) with the surrounding environment point cloud Xs. Then, we augment this combined point cloud with one extra 1D arrays of ones and zeros, to indicate if point belongs to X; or Xs. along the XYZ dimension form tensor of shape (M + M, 1). With this augmented input, a straight- forward method for stable classification would be using PointNet++ [26]. However, we find that
5

---

this naive approach does not provide satisfying results due to the intrinsic difficulty of analyzing stability among two separate geometry components.
To facilitate learning, we propose to solve this problem via a multi-task supervised learning ap- proach. After obtaining the feature from the PointNet++ backbone, we have three task branches that perform stable classification, contact classification, and contact point regression simultancously. For contact point regression, we use a PointNet++ decoder to regress the 3D offset v; from each point p; of the input point cloud to its nearest ground-truth contact point with a loss defined as:
Loffeet
Σ L(p + Us arg min ||c -p-l2) P.E
(2)
oEC
Here X denotes the input point clouds including the transformed object point cloud Tz (X,) and the environment point cloud Xs. Based on the predicted Ta, pose, we calculate the nearest ground truth stable pose from Eqn. and collect the corresponding contact point sets between the object and the environment in the simulation denoted C. We implement L smooth L1 loss. We also adopt variance loss Leariance similar to [27] to reduce the variance of the group {p; (p;+t22 that correspond to the same contact point. We train two MLP classifiers for stable pose and contact classification with losses denoted as Latable and Leontact respectively. Our total loss is defined as:
Lc = Lsaable + foft + Xe + A Agaariiacce
(3)
where Lstable and Lcontact are standard binary cross entropy losses.
Leveraging the above stable pose classifier function denoted as V, we apply CEM [25] to search for a locally optimal goal pose starting from the initial predicted poses. We first apply the predicted transformation T to the object point cloud X; to get point cloud X,, A point a in the CEM search- ing action space is a 6D transformation To which transforms the object point cloud X; into Ta (X.). A point in the CEM state space is the transformed object point cloud along with the surround- ing environment point cloud (Ta (X,,),,.ss). Xs). When selecting the action a, we run a derivative-free optimization method CEM [25] to search within the 6D pose space to find a 6D transformation Ta associated with the highest score in the value model V(s).
a" = arg max V(Ta (X,), Xs)
(4)
The 6D pose associated with a" is the final stable placement pose produced by model.
4.3 Object Grasping
In this subsection, we describe how our system performs valid grasping operations. Recall that a grasp pose 9% of the object O is parameterized by two contact points on the object surface (Pi. q and an approaching direction of the robotic hand di.- We adopt UniGrasp [6], an efficient data-driven grasp synthesis method, to generate list of valid grasp poses {G]} of the object O,, UniGrasp learns a deep neural network to select a set of contact points (p), q ) from the input point cloud of the object O. We refer to Shao et al. [6] for more detailed description of the method. Given two contact points Pjp and qi. we calculate a plane perpendicular to the line segment of the two contact points intersecting at the middle point of the line segment. We then consider 36 approaching directions evenly distributed within the plane, which produces 36 potential grasp poses. For each grasp pose Gi we solve the inverse kinematics of the robot. If there is no valid inverse kinematic solution or there are collisions in all possible approaching directions, we drop the contact point pair and continue checking for the next pair of contact points. We keep running this process until we find a valid grasping operation if there exists one.
5 Experiments
In this work, we develop a learning framework for robots to learn to regrasp objects. Our experi- ments focus evaluating the following questions: (1) How effective is our proposed learning-tor place approach compared to other baselines? (2) Whether our proposed learning-to-regrayy to-rerasp pipeline can find good solutions to the object regrasping problem? (3) Whether the multi-task learning for contact point predictions we use in Sec. 4.2 is useful or not? (4) Whether our proposed pipeline is robust to various sources of noises (e.g. object segmentation, sensing, dynamies)? Please refer to supplementary materials for experiments associated with (3) and (4), ablation study, and failure cases analysis.

---

Dataset. We construct a dataset containing 50 objects (i.e., spoon, fork, hammer, wrench, etc.) and 30 supporting items (i.e., mug. box, bowl). We then split these objects and supporting items into training and test sets. No test objects and supporting items have been seen during training. We generate 249 pairs from the training set of objects and supporting items, and 38 pairs from the test. We collect the placement data for supporting items by randomly placing the supporting items with respect to the initial environment (i.e., the ground), and running the simulation to check whether the supporting items keep static. To make the stable placement robust to variant dynamics and geometry, we randomize the dynamic parameters including friction, mass, and external forces. We describe the details in the supplementary. In total, we generate around one million poses for training set and 15 thousand poses for testing set. We visualize the data in supplementary material.
Settings. We first set up the simulation environment in PyBullet [28] for evaluating pipeline on synthetic data. We load the object at the predicted pose and check if there is collision and whether the object is stable. We then evaluate the regrasping task on real robot system.
full
5.1
Object Stable Placement
We evaluate our proposed framework on learning stable placement poses in this subsection.
method
bowl- fork
spoon
bowl-
hammer
box-
wrench
box-
spatula
box-
mug-
fork
spoon
mug-
corkserew
mug-
mean
Random
0.000
0.001
0.109
0.039
0.000
0.000
0.000
0.012
0.020
Jiang et [12]
0.037
0.080
0.201
0.113
0.047
0.018
0.125
0.092
0.089
Jiang et [12] + heuristic
0.175
0.041
0.437
0.852
0.603
0.013
0.128
0,000
0.281
Ours
0.869
0.848
0.894
0.911
0.905
0.674
0.834
0.829
0.845
Table 1: Overall comparison of the stable placement accuracy. We report the accuracy of our method and different bascline methods for each category and the average accuracy over all the categories. Our method significantly outperforms these baseline methods.
method
bowl-
fork
spoon
bowl-
hammeer
box-
wrench
box-
spatula
box-
mug-
fork
mug- spoon
mug- corkscrew
mean
Jiang et al. [12] + hearlstic
2.500
1,000
28.000
28.333
10.333
2,000
4,250
0.000
9,552
Ours
16.900
18.100
45.850
49.100
52.133
14,500
15.750
12,450
28.098
Table 2: Overall comparison of the predicted stable placement poses' diversity.
Evaluation of Object Placement Accuracy. Our pipeline samples and refines 128 poses for each ob- ject and supporting item pair. We use the pose clas- sifier to estimate the scores for these 128 poses and use threshold of 0.8 to filter out the poses with low scores. To verify the remaining poses, we run for- ward simulation in PyBullet [28] to check whether the objects these poses stable. The results of the sta- ble placement accuracy for each category are reported in Table 1.
0.61
0.74
0.87
0.02
0.79
0.89
0.93
0.9
We first adopt a baseline (Random) which is based on a random pose sampling strategy. The rotation value is sampled from SO(3) uniformly, and translation value is uniformly sampled within this region of [=:0.05, +0.05, 2+0.02] in the supporting item's local frame, where z is determined by the maximal height of each supporting item. We use the meter as the unit of trans- lation value throughout the experiment section.
Figure 4: Point cloud visualization of sam- pled stable poses from our proposal net- work. Our proposal network can generate diverse stable placements of the candidate object (colored green). We also mark the predicted stable scores.
We slightly modify the approach in [12] to work in our setting and use it as another baseline ( (Jiang et al. [12]), in which a stable pose classifier is learned based on hand crafted features. We randomly sample 128 poses in the local frame of the supporting environment, and classify these poses using the stable pose classifier. We evaluate the accuracy of these stable poses in PyBullet [28] and report the results in Table 1. The results suggest that even with random sampled poses, a simple classifier with hand-craft feature can improve the stable placement accuracy. We also design another baseline that adopts heuristic rules for pose sampling but with the same classifier from [12]. For this baseline ( (Jiang et al. [12] + heuristic), the object translation in the horizontal dimensions (XY dimensions)
7

---

Figure 5: Demonstration of regrasping tasks on real world data and visualization of stable poses generated by our model.
sampled in the radius of 0.9 = r and the translation in the gravity dimension (Z dimension) is determined by 0.01, where and are the radius and maximal height of supporting item respec- tively. We additionally rotate the object to make the longest axis of the object horizontally. While this baseline can achieve some reasonable results in easy situations when the supporting item has a nearly flat horizontal surface (i.e., box- wrench, box-spatula), perform worse with supporting items with complicated geometry (i.e., bowl, mug). The results demonstrate that good pose pro- posal algorithm is also of great importance to the success of stable object placement. With carefully designed stable pose proposal network and stable pose classifier network, framework achieves the best results over all the approaches.
Evaluation of Object Placement Diversity. We evaluate whether the pose proposal network can generate a reasonable distribution of the stable poses for placing the object over the supporting items. A diversifying stable pose distribution is crucial for the success of finding desirable regrasping solutions. For each input object and supporting item pair, we draw 128 poses from the proposal network, and transform the candidate object to the proposed stable poses. The qualitative results are presented in Fig. 4. Our visualization demonstrates that our proposed stable poses yield diverse object placements.
To quantitatively measure the diversity of the stable object placements, we define two poses as two different poses if their translation difference is larger than and the rotation angle between them is larger than 72. In our experiment, 71 and T2 are set to be 0.03m and 30 respectively. We report the average number of different stable object placements in Table 2. The results showing that although random uniform sampling combined with heuristic rules can generate diverse stable poses for some easier environment (i.c., boxes with flattened surface), it can not handle complicated geometry. Our proposed method outperforms the baseline by a large margin.
5.2 Object Regrasping Performance
We evaluate the performance of our model in real-world experiments on everyday objects. We first test the pose proposal network on point clouds from Intel RealSense RGB-D camera. Utilizing the camera matrix, we can reconstruct 3D point cloud. The point clouds are fed into our model to predict stable placement poses. We then run Unigrasp [6] to generate a list of valid grasp poses and search for the desired target poses based on the pipeline described in Sec. 4. The generated stable placements and several key frames of the regrasping process are visualized in Fig 5. For more results on the regrasping experiments, please refer to our supplementary material.
Conclusion
We present a system that can regrasp a diverse set of everyday objects to desired grasp poses for various manipulation tasks. Our system learns to predict the stable placement of objects based on partial point clouds of the object and the surrounding environments. The system also creates regrasp graph and allows robots to leverage and actively change other objects' poses to provide extra supports. We demonstrate the effectiveness of our system on both challenging synthetic dataset and in real world regrasping tasks. Currently our system only takes raw vision sensory data as inputs. For future work, we would like to explore how to incorporate other modalities such as force or tactile signals in our system.
8

---

Acknowledgments
We would like to thank the UnitX company (https://www.nitxlaft) com) for providing us the URS robot arm used in the real robot experiments. We thank Shenli Yuan for helping us set up the cus- tomized two-fingered gripper, and Hongzhuo Liang for helping set up the network configuration.
References
[1] Tournassoud, Lozano-Perez, and E. Mazer. Regrasping. In Proceedings. 1987 IEEE International Conference on Robotics and Automation, volume 4, pages 1924-1928, 1987.
[2] A. Sahbani, S. El-Khoury, and P. Bidaud. An overview of 3d object grasp synthesis al-
gorithms. Robotics and Autonomous Systems, 60(3):326-336, 2012. ISSN 0921-8890. http:://bb..oog////1.00111o..... sci occaacc/articae////////811011110
URL https: http://www.ceccciieetttttoo com/ Autonomous Grasping.
[3] Mahler, J. Liang. Niyaz, M. Laskey, Doan, X. Liu, A. Ojea, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. 2017.
[4] J. Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, S. McKinley, and K. Goldberg. Learning ambidextrous robot grasping policies. Science Robotics, 4(26))eaau4984, 2019.
[5] I. Lenz, H. Lee, and A. Saxena. Deep learning for detecting robotic grasps. The International Journal of Robotics Research, 34(4-5):705 7224,, 2015. do:110.177707786644444559777.. URL https:/ //ooi org/10. 1177 1177717777364994444999609
[6] L. Shao, F. Ferreira, M. Jorda, V. Nambiar, J. Luo, E. Solowjow, J. A. Ojea, O. Khatib, and
J. Bohg. Unigrasp: Learning a unified model grasp with multifingered robotic hands. IEEE Robotics and Automation Letters, 5(2):2286-22933 2020.
[7] Bohg, Morales, Asfour, and D. Kragic. Data-driven grasp synthesis a survey, IEEE Transactions on Robotics, 30(2):289-309 April 2014. ISSN 1552-3098. Noi:10.1109/TRB 2013.2289018.
[8] T. LLzzao--ééeee J. L. Jones, P. A. O' O'Donnell, and E. Mazer. Handey: A Robot Task Planner. MIT Press, Cambridge, MA, USA, 1992. ISBN 0262121727.
[9] F. Rohrdanz and F. Wahl. Generating and evaluating regrasp operations. In Proceedings of International Conference on Robotics and Automation, volume 3, pages 2013-2018 vol.3, 1997. 1o
[10] S. Stoeter, S. Voss, N. Papanikolopoulos, and H. Mosemann. Planning of regrasp opera- tions. In Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36228CC,, volume 1, pages 245-250 vol.1, 1999. 1oi: 199:00011990000001999999999999
[11] W. Wan, H. Igawa, K. Harada, Onda, Nagata, and N. Yamanobe. regrasp planning component for object reorientation. Autonomous Robots,
2019.
[12] Y. Jiang, M. Lim, C. Zheng, and A. Saxena. Learning to place new objects in a scene. The International Journal of Robotics Research, 3109):1021-1043, 2012. 2012.
[13] A. Sahbani, S. El-Khoury, and P. Bidaud. An overview of 3d object grasp synthesis algorithms. Robotics and Autonomous Systems, 60(3):326 336, 2012. ISSN 0921-8890. Autonomous Grasping.
[14] K. Harada, T. Tsuji, K. Nagata, N. Yamanobe, and H. Onda. Validating an object placement planner for robotic pick-and-place tasks. Robotics and Autonomous Systems, 62(10):1463- 1477, 2014. ISSN 0991-8890. www.scciencedired.. www/science///reeeeeiccciceeeaa htt.a/acww...ooo/ccccc/////ttttt
URI. https:

---

[15] J. A. Haustein, K. Hang, J. Stork, and D. Kragic. Object placement planning and optimization for robot manipulators. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7417-7424, 2019. (00:00011000RR55444222219999.
[16] T. Lozano-Pérez and L. P. Kaelbling. A constraint-based method for solving sequential manip- ulation planning problems. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3684-3691, 2014.
[17] M. Raessa, W. Wan, and K. Harada. Planning to repose long and heavy objects considering combination of regrasp and constrained drooping. Assembly Automation, 2021.
[18] C. Cao, W. Wan, J. Pan, and K. Harada. Analyzing the utility of a support pin in sequential robotic manipulation. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages -5044 IEEE, 2016.
[19] J. Ma, W. Wan, K. Harada, Q. Zhu, and H. Liu. Regrasp planning using stable object poses sup- ported by complex structures. IEEE Transactions on Cognitive and Developmental Systems, 11(2):277-26, 2018.
[20] Z. Xue, J. M. Zoellner, and R. Dillmann. Planning regrasp operations for multifingered robotic hand. In 2008 IEEE International Conference on Automation Science and Engineering, 100.///OE....0.00000000
pages 778-783, 2008
[21] N. Chavan-Dafie and A. Rodriguez. Regrasping by fixtureless fixturing. In 2018 IEEE 14th International Conference on Automation Science and Engineering (CASE), pages 122-129, 2018. doi:10. 100000000AAAAE220..55500000
[22] S. Yuan, L. Shao, C. Yako, A. Gruebele, and K. Salisbury. Design and control of roller grasper v2 for in-hand manipulation. In 2020 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), 2020.
[23] Cruciani, K. Hang, C. Smith, and D. Kragic. Dual-arm in-hand manipulation and regrasping using dexterous manipulation graphs. arXiv preprint 1rXiv:1904.11382. 2019.
[24] J.PP. Saut, M. Gharbi, J. Cortés, D. Sidobre, and T. Siméon. Planning pick-and- placce tasks with two-hand regrasping. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4528-4533. IEEE, 2010.
[25] R. Y. Rubinstein and D. Kroese. The Cross Entropy Method: Unified Approach To Combi- natorial Optimization, Monte- Carlo Simulation (Information Science and Statistics). Springer- Verlag, Berlin, Heidelberg, 2004. ISBN 038721240X.
[26] C. R. Qi, L. Yi, H. Su, and I.. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. arXiv preprint arXiv: 1706.02413, 2017.
[27] L. Shao, P. Shah, V. Dwaracherla, and J. Bohg. Motion-based object segmentation based on dense rgb-d scene flow. IEEE Robotics and Automation Letters, 3(4):3797-3800) Oct 2018. ISSN 2377-3766. doi: 00.100010991RRA22188225555
[28) E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning.
org, 2016-2019.
10


Reviews:
[{"review_id": "y_AkQUM4bSp", "reviewer": "Reviewer_nUvc", "summary": "In this paper, the robot's goal is to achieve a desired grasp on an object by constructing a sequence of pick-and-place operations, i.e. a form of regrasping. The paper proposes a novel stable object pose prediction model with deep-learned features extracted directly from raw point clouds. This is used to generate a diverse set of placement poses for a regrasping graph, which is then searched over to find an action sequence. The regrasping graph is constructed of nodes that correspond to stable poses for all of the objects in the scene and edges that correspond to a pick-and-place operation that moves one object. Thus the pose prediction method proposed by the paper is used to generate the nodes, the edges are generated via motion planning, existing work (UniGrasp) is used to generate the grasps, and depth first search is used to find a path through the graph.\n\nThe stable object pose prediction framework is a two-part process that first predicts a diverse set of \"rough\" poses that are then refined via the second stage. The first stage passes partial point clouds into a PointNet++ encoder, along with a Gaussian noise vector to encourage the diverse solutions. A multilayer perceptron outputs a 6-DoF object pose that is used to transform the object's point cloud for the second stage. Rather than directly predicting stability, the paper frames this question as a multi-task supervised learning problem wherein the network branches to predict stability classification, contact classification and contact point regression. The method leverages this classifier and CEM to further refine the 6-DoF stable pose of the object. For training data the paper presents a dataset of pairwise stable placements evaluated in PyBullet. Within the main paper, the framework is evaluated against a baseline strategy that randomly samples placements. With respect to the validation of the stable object pose prediction framework, the experiments first evaluate its ability to generate diverse poses (the first stage) and then evaluate whether the generated poses are stable using PyBullet (the second stage). The entire system is then evaluated, with some ablation studies, across 30 scenes in simulation.\n\nThe appendix contains another ablation study, discussions of the collection of real-world point clouds and comparisons to two other baselines from existing work. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper's primary contribution is a novel stable object pose prediction method that operates directly on point clouds. This method is used within a pick-and-place regrasping framework. The paper is clear in stating that the pose prediction method is the main focus and is fairly clear that the regrasping graph structure draws heavily from prior work (which the paper does a good job reviewing over). The problem statement (Section 3) very precisely defines the terminology and problem being tackled. The generality of a pose prediction method that operates directly on point clouds means that this could be broadly applied in other frameworks and applications. \n\nThe random sampling strategy baseline that the method is compared against is a very weak baseline, as evidenced in Table 1 by the fact that it often scores 0% (or close to 0%) accuracy. With such low performance, which is to be expected from such a method, it's unclear if it actually serves as a useful comparison. The appendix provides much more reasonable baselines to compare against. \n\nA main strength of the method, as discussed above, is that it operates directly on point clouds, which somewhat frees the method from being as object-centric. The creation of the dataset, as described in Sec 5, seems to suggest that the method only considers placements where the object is supported by one object. Would augmenting the dataset allow for placements where the object is supported by multiple objects? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am weakly recommending the acceptance of this paper. The proposed method to generate stable object poses from point cloud data is sound and has wide applicability. The regrasping aspect of the paper is not overly novel and instead serves as a domain where the pose prediction can be compellingly demonstrated. As mentioned above, the experimental section is weakened by the poor baseline but this can be easily overcome by highlighting the comparisons made in the appendix. \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "My primary suggestion would be to include the baseline comparisons from the appendix (i.e. the two variants of Jiang et al) into the main paper. These comparisons greatly strengthen the experimental section and many readers could miss them if they are exclusively in the appendix.\n\nA few more minor questions/comments:\n- The entire regrasping system is described as having a 73.3% success rate. When the system fails, where are these failures normally coming from? Are poor placements the primary source of failure or is there another aspect?\n- The regrasping shown in the video attachment is primarily performed by a human. A more compelling demonstration of the system would include simulation videos of the robot regrasping from the experiments described in Sec 5.2.\n- In reviewing the literature on regrasping, it would be worth mentioning other formulations such as in-hand and two-hand regrasping [1, 2, 3, 4].\n- With respect to future iterations of the system, it could improve efficiency to incorporate lazy search [5, 6] in the construction and search of the regrasp graph. I would not expect a revised version of this paper to include this modification.  \n- The regrasping problem in this paper is framed such that there is one target object and any other objects in the scene can be moved in order to accommodate the target object. How could the method be adapted to handle multiple target objects? \n\n\n[1] Cruciani, Silvia, et al. \"Dual-arm in-hand manipulation and regrasping using dexterous manipulation graphs.\" arXiv, 2019.\n\n[2] Chavan-Dafie, Nikhil, and Alberto Rodriguez. \"Regrasping by fixtureless fixturing.\" IEEE CASE, 2018.\n\n[3] Xue, Zhixing, J. Marius Zoellner, and Ruediger Dillmann. \"Planning regrasp operations for a multifingered robotic hand.\" IEEE CASE, 2008.\n\n[4] Saut, Jean-Philippe, et al. \"Planning pick-and-place tasks with two-hand regrasping.\" IEEE IROS, 2010. \n\n[5] Dellin, Christopher, and Siddhartha Srinivasa. \"A unifying formalism for shortest path problems with expensive edge evaluations via lazy best-first search over paths with edge selectors.\" ICAPS, 2016.\n\n[6] Bohlin, Robert, and Lydia E. Kavraki. \"Path planning using lazy PRM.\" IEEE ICRA, 2000. "}}, {"review_id": "yJvYtvShuhw", "reviewer": "Reviewer_FiyG", "summary": "The paper proposes a method to plan sequences of regrasps by learning a pose stability prediction network from partial point cloud observations. Separately, a stable pose proposal network is also learned. Together with an off-the-shelf grasp sampler (Unigrasp), a graph of valid regrasp (grasp and placement poses) can then be constructed. Searching in this graph can generate a sequence of regrasp actions that move an object to a desired target pose. Simulation experiments show a regrasp success rate of 73.3% across 9 object categories, while a baseline using randomly proposed poses achieves 0%.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-\tUsing partial pointclouds as input is an improvement over prior works that require full object information.\n-\tUse of multi-task learning to improve pose stability classification is interesting.\n-\tThe paper is clearly written and easily understood.\n\nWeaknesses\n-\tI have two main criticisms. One is there does not seem to be a notion of robustness in the stable pose prediction dataset. Two is it was not clear what kind of generalization was being tested aside from the test object pairs.\n-\tFor robustness – some of the visualized stable poses appear quite precarious, with a spoon being placed near the edge of the top of the bowl. It seems quite possible that with small noises in sensing and actuation that some of the predicted stable poses would yield unstable behaviors when executed by a robot. Stable pose data could benefit from perturbations in poses during data generation. Additional noises in object dynamics (friction, mass, and center of mass) and sensing (it was not mentioned whether the partial point clouds were rendered with realistic noise profiles) during data generation would make the method more compelling.\n-\tFor generalization – although pairs of objects were tested, it was not clear whether the pair contained objects that were used in training. For example, if bowl-fork is a test pair, does that mean both bowl and fork were not in the training set, or does that mean the bowl-fork pairing was not in the training set but the objects themselves appeared in other pairings in the training set?\n-\tThe latter case would make the paper more compelling, and additional experiments on generalizing to completely unseen objects, not just object pairs would be appreciated.\n-\tWhile the paper mentions using partial pointclouds, in Figure 3 the pointcloud appears to show part of the spoon is visible even though it should be occluded by the front side of the bowl. Is this just for visualization? In reality, if the camera is placed at that angle, then the lower half of the spoon would be occluded by the bowl, so that part of the pointcloud should not be visible.\n-\tThis work also seems to assume that either the objects are completely separated in the beginning so their pointclouds can be obtained separately, or the method has access to a robust pointcloud part segmentation algorithm that can do the same when the objects are close together or in contact. However, there is no discussion of such assumptions in the paper.\n-\tThe video does not show a complete sequence of a robot executing the planned regrasp motions. Having such a video demonstration would more clearly illustrate the proposed method, even if it’s only in simulation.\n-\tThe experiment results are a little hard to interpret. The authors mainly report pose stability prediction accuracies, but not the success rate of the overall regrasping framework. Only an average success rate of 73.3% is given, but more detailed results and analysis here would strengthen the paper.\n-\tThere is also no in-depth discussion of failure modes – are there any patterns to the cases when stability prediction fails? Why does the model perform poorly in some object pairings but not others?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation of weak reject is based on the lack of more in-depth discussions of the problem and presentation of regrasp results.\n\n**Update** I have updated my recommendation to weak accept after the authors' additional experiments and clarifications.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see the Strengths and Weaknesses section. The website was also empty at the time of review.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "mLRZEOZ2Ndi", "reviewer": "Reviewer_bwfS", "summary": "The paper addresses the problem of placing a target object in a desired pose using a robotic arm, potentially requiring a sequence of grasp and place (regrasp) manipulation of the target object or other supporting objects in the scene. The proposed method employs a leaned module that predicts stable placements of an object in its surrounding, both expressed as point clouds. The placement module generates possible poses which are evaluated by stable, contact, and offset from contact points, and the CEM method is used for selection. A sequence of pick and place is planned by constructing  a regrasp graph by applying grasp and place motions, connecting different nodes corresponding to the stable poses of all the objects in the scene that are accessible through a pick and place of exactly one object, and searching for a path from the initial node to a target node using depth first search. The  method is validated in simulation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I find the method quite interesting and creative. The problem is very challenging and well motivated. The reported results for both stable placement and overall planned sequence in simulation are also appealing.\nHowever, the proposed method makes some assumptions and is applied in simulation without really reasoning about how this could work in real settings and not just simulation.\nThe assumption that a perfect segmentation of the scene is provided at each point from which it can be derived which are the objects and which is the target is often not very realistic. I would try to suggest to at least show the robustness of the method to some level of uncertainty.\nThe data used for stable placement with the accompanied ground truth annotations would be difficult to replicate in real setting. If manual annotations are required or reliance on sim2real capabilities, this should be somehow considered in the paper.\nIn addition, the regrasp construction and search, which is executed in real-time, could potentially be very computationally demanding. Without any guiding heuristic, the graph can grow very large until the actual target is achieved, and there are no convergence guarantees, at least based on how this process is described in the paper. I think some notion of computation times (construction of nodes and edges) and reasoning about bounds on the grasp size are due. In addition, I would consider incorporating leaning also in this stage as well.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper describes an interesting approach for solving a challenging and important problem. I think it would be much better after providing some indication about how applicable it would be in real settings", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "All issues are described above", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers are mixed. Whilst all reviewers are encouraged by the novelty over prior works, two reviewers raise various issues regarding how applicable this method would be in real-world settings, and I would like to hear the authors' opinions on these comments. Whilst real-world experiments are not always essential to demonstrate the importance of a method, some analysis on the robustness to real-world noise, such as segmentation masks, is important to at least show potential for this to work in the real world. Reviewer bwfS also notes how the annotations rely on ground-truth data, which would be difficult to obtain in a real-world setting, and yet sim-to-real experiments are not provided. So, as well as answering queries raised by the reviewers, I would like to see the authors' discuss how applicable the method is to real-world settings, and how they can prove this applicability.

-----

Following the reviews, the authors have now provided evidence of applicability to the real world, with additional real-world experiments and analysis of sensitivity to real-world noise. Authors have also addressed many of the issues raised by the reviewers, and all three reviewers now recommend acceptance of the paper.
Decision: Accept (Poster)