V-MAO: Generative Modeling for Multi-Arm Manipulation of Articulated Objects
Authors: Xingyu Liu
Kris M. Kitani
Venue: CoRL2021 Poster (2021)
Keywords: Articulated Object
Generative Model
Variational Inference

Abstract: Manipulating articulated objects requires multiple robot arms in general. It is challenging to enable multiple robot arms to collaboratively complete manipulation tasks on articulated objects. In this paper, we present V-MAO, a framework for learning multi-arm manipulation of articulated objects. Our framework includes a variational generative model that learns contact point distribution over object rigid parts for each robot arm. The training signal is obtained from interaction with the simulation environment which is enabled by planning and a novel formulation of object-centric control for articulated objects. We deploy our framework in a customized MuJoCo simulation environment and demonstrate that our framework achieves a high success rate on six different objects and two different robots. We also show that generative modeling can effectively learn the contact point distribution on articulated objects.

V-MAO: Generative Modeling for Multi-Arm Manipulation of Articulated Objects
Xingyu Liu Robotics Institute Carnegie Mellon University xingyul3@ca cmu edu
Kris M. Kitani Robotics Institute Carnegie Mellon University kkit ani (cs. cmu. edu
Abstract: Manipulating articulated objects requires multiple robot arms in gen- eral. It is challenging to enable multiple robot arms to collaboratively complete manipulation tasks articulated objects. In this paper, present V-MAO, framework for learning multi-arm manipulation of articulated objects. Our frame- work includes variational generative model that learns contact point distribution over object rigid parts for each robot arm. The training signal is obtained from interaction with the simulation environment which is enabled by planning and a novel formulation of object-centric control for articulated objects. We deploy our framework in customized MuJoCo simulation environment and demonstrate that our framework achieves a high success rate on six different objects and two dif- ferent robots. We also show that generative modeling can effectively learn the contact point distribution on articulated objects.
Keywords: Articulated object, generative model, variational inference
1 Introduction
In robotics, one of the core research problems is how to endow robots with the ability to manipulate objects of various geometry and kinematics. Compared rigid objects, articulated objects contain multiple rigid parts that are kinematically linked via mechanical joints. Because of the rich function- ality due to joint kinematics, articulated objects used in many applications. Examples include opening doors, picking objects with pliers, and cutting with scissors, etc.
While the manipulation of a known rigid object has been well studied in the literature, the manipu- lation of articulated objects still remains challenging problem. Suppose an articulated object has N rigid parts. In the most general case, it requires N robot grippers to manipulate the N object parts where each gripper grasps one rigid part to fully control all configurations of the articulated object, including position and joint angles How teach the robot arms to collaborate safely without collision is still a research problem [1, 2]. Our intuition that, instead of a deterministic model, sampling from distribution of grasping actions can provide more options and can increase the chance of finding feasible collaborative manipulation after motion planning. Therefore, the gen- erative modeling of grasping can be a solution to the problem.
We hypothesize that an object-centric representation contact point distribution contains full in- formation about possible grasps and can generalize better across different objects. Therefore it is preferred over robot-centric representation such as low-level torque actions, due to the uncertainty of robot configurations. Specifically for articulated objects, further hypothesize that the grasp- ing distribution on one rigid part is conditioned on the geometric and kinematics feature as well as grasping actions other parts, and can be learned from interacting with the environment.
In this paper, we propose a framework named V-MAO for learning manipulation of articulated ob- jects with multiple robot arms based on an object-centric latent generative model for learning grasp- ing distribution. The latent generative model is formulated as conditional variational encoder
'Though certain scenarios, still possible that an articulated object manipulated by single grip- per, c.g. dexterous manipulation of a pair of scissors with one human hand. However, such manipulation is extremely complex and is not scalable for other arbitrary articulated objects.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

RGBD Observation
Samples of Generated Contact Point Probability Map
Grasp & Manipulation
Figure 1:
Generative Modeling of Multi-Arm Manipulation of Articulated Objects. Given the RGBD scan observation of the articulated object and the scene, we use a generative model to learn the distribution of feasible contacts of each robot arm. The generative model for the second robot arm (second row) is conditioned on the action performed by the first robot arm (first row).
(CVAE), where the distribution contact probability on the 3D point clouds of one object rigid part is modeled by variational inference. Note that the contact probability distribution on one object part conditioned on the grasping actions already executed on other object parts which can observed from the current state of 3D geometric and kinematics features. To obtain enough data for training, our framework enables automatic collection training data using the exploration of contact points and interaction with the environment. To enable interaction, we propose a formulation of Object- centric Control for Articuated Objects (OCAO) to move the articulated parts to desired poses and joint angles after successful grasp. The framework is illustrated in Figure 1.
We demonstrate our V-MAO framework in a simulation environment constructed with MuJoCo physics engine [3]. We evaluate V-MAO on six articulated objects from PartNet dataset [4] and two different robots. V-MAO achieves over 80% success rate with Sawyer robot and over 70% success rate with Panda robot. Experiment results also show that the proposed variational generative model can effectively learn the distribution of successful contacts. The proposed model also shows advantage over the deterministic model baseline in terms of success rate and the ability to deal with variations of environments.
The contributions of this work are three-fold:
1. A latent generative model for learning manipulation contact. The model extracts articulated object geometric and kinematic representations based on 3D features. The model implemented based conditional variational model.
A mechanism for automatic contact label generation from robot interaction with the environment. We also propose formulation of Object-centric Control for Articuated Objects (OCAO) that enables the interaction by the controllable moving of articulated object parts.
3. We construct a customized MuJoCo simulation environment and demonstrate our framework on six articulated objects from PartNet and two different robots. The proposed model shows advantage over deterministic model baseline in terms of success rate and dealing with environment variations.
2 Related Works
Generative Modeling of Robotic Manipulation. Previous work has proposed to leverage latent generative models such variational auto-encoder (VAE) [5] and generative adversarial network (GAN) [6] in various aspects of robotic manipulation. Wang et al. [7] proposed to use Causal InfoGAN [8] learn to generate goal-directed object manipulation plan directly from raw images. Morrison et al. [9] proposed generative convolutional neural net (CNN) for synthesizing grasp from depth images. Mousavian al. [10] use VAE model the distribution of pose of 6D robot grasp pose. Other works have proposed to use generative model to learn more complex grasping tasks such as multi-finger grasping [11] and dexterous grasping [12, 13]. Generative modeling can also be seen in other aspects of robotic manipulation such as for object pose estimation [14] and robot gripper design [15]. Our approach focuses on using variational latent generative model on 3D point clouds for learning object contact point distributions on articulated objects.

---

Earoser Newwor Modale
Devaly Netword Mistal
AGBD Inpor
Inerre
Colored 3D Fone Cloud
Mexunstracted Claniae! Prod Map
Operational Space Control
Rigid Part
o7)
Estimation
Contert Map Label
Object-ceniee centric Contral
Manipulation Step
aGRD Aepurt
Ioverse Kinemanics
Colored 3D Pore Cloud
Revonsmmcted Contact Pros Map
Operational Space Control
Rigid Part Pose Estimation
Ma, ofj
Objact-centric Counol
Contart Proh Map Label
Manipulation Step 2
Figure 2: Architecture of V-MAO Framework. Given the scene point cloud, each step of the manipulation modeled by a conditional VAE. Interaction with the object is done through inverse kinematics and object-centric control. Given the manipulation of one object part with robot is successful, V-MAO will move the manipulation of the next object part with the next robot.
Articulated Object Manipulation. The task of manipulating articulated object has been studied on both perception and action perspective in the literature. For example, Katz et al. [16] proposed relational state representation of object articulation for learning manipulation skill. From interac- tive perception, Katz et al. [17] learns kinematic model of an unknown articulated object through interaction with the object. Kumar et al. [18] propose to estimate mass distribution of articulated objects through manipulation. Narayanan et al. [19] introduced kinematic graph representation and an efficient planning algorithm for object articulation manipulation. More recently, Mittal et [20] constructed mobile manipulation platform that integrates articulation perception and planning. Our work models contact point distributions from object geometry using a generative model and propose an object-centric control formulation for manipulating articulated objects.
Multi-Arm in Robotic Manipulation. Using multiple robot arms in manipulation tasks without collision is challenging problem. Previous works have investigated using decentralized motion planner avoid collision in multi-arm robot system [1]. Previous works have also investigated using multiple robots in the application of construction assembly [2], pick-and-place [21], ], and table- top rearrangement [22]. Our work is an application of using multiple robot arms manipulate articulated objects where we focus on learning the contact point distribution on object parts.
Method
Our V-MAO framework consists of: 1) variational generative model for learning contact dis- tribution on each object rigd part (Section 3.2); and 2) inverse kinematics, planning, and control algorithms for interacting with the articulated object (Section 3.3). In Section 3.1, we first formally define the problem using the generative model articulated object manipulation. In Section 3.4, we provide additional details to our framework. The framework is illustrated in Figure 2.

---

3.1 Problem Definition
We consider the problem of multi-arm manipulation of articulated objects. The object has N rigid parts and the system contains N robot arms where N 2 2. Robot arm i executes high-level grasping action Ii and moving actions a, on the objects. Grasping action 9i is represented by the contact points on object part i. Moving actions d; send torque signals to robot to move object part to desired 6D pose Ti- The whole manipulation is sequence of 9i and aç.
Given colored 3D point clouds from RGBD sensors, suppose P; [xx, E RR+C, -j= 1...., nz} is the 3D point cloud of the i-th rigid part of the object, where n. is the number of points and C the number of feature channels, e.g. color values and motion vectors. Suppose Ps = {xx (8) E IR+C}
is the 3D point cloud of the non-object scene and ()) is the 6D pose of i-th rigid object part after executing grasping actions I1, 92, .........h Ok- Note that the complete 31) point cloud of the scene (k)) = Ps U(U P.) = o Mt II, 7hh) a function of grasp actions (91, $2. ................................ ....)...)))))))..).......)) 9h)) robot
and object part poses ( where O is the observation of the environment and depends forward kinematics. We assume there is no cycle in object joint connections so that there is no cyclic dependency among object parts. We formulate the manipulation problem learning the following joint distribution of (g1, 92, ................................ ...) given initial scene point cloud (0):
P(a....., On |p (0)) = (or |p P2 ((
... P.(. |
(1)
P. is the contact point distribution model of i-th rigid part. Equation (1), we use conditional probability to decompose P and assume previous point clouds have no effect on the current grasping distribution. Our goal is to learn distribution P; for each object part that covers the feasible grasping as widely possible so that sampled grasping actions gi and subsequent moving actions can successfully take all object parts to desired poses Tí
3.2 Generative Modeling of Contact Probability
Our generative model is based on variational inference [5]. Different from 2D images where the positions of the pixels are fixed and only the pixel value distribution needs be modeled, in 3D point clouds, both geometry and point feature distribution need to be modeled. Therefore, instead of directly using multi-layer perceptions (MLPs) on pixel values similar vanilla VAE, our model needs to learn both local and global point features in hierarchical fashion [23]. Moreover, in our model, the reconstructed feature map should not only be conditioned latent code, but hicrarchical geometric features as well.
Encoder and Decoder for One Single Step. Given colored 3D point cloud appended with suc- cessful grasping probability maps p = {x, E R3+4] where four feature channels are RGB and probability values, our model split the feature channels to construct two point clouds appended with different features: a 3D color map Prah = {x, E RR+3) as the input to geometric learning, and P. = (x, E TR 3+ as the 3D contact probability map to reconstruct using generative model.
An encoder network Fe(:; 0c) is used to approximate the posterior distribution q(= Pes Prgb) of the latent code z. Another encoder network Frgo (:; (igh) learns the hierarchical geometric representation and outputs features h1, h2, h& at different levels:
[n, o| FP Pigsi 0c), [h,, h2, hy] = (Prgbi Orgb)
where y and are the predicted mean and standard deviation of the multivariate Gaussian distribu- tion respectively. The latent code z is then sampled from the distribution and is used to reconstruct the contact probability map on point clouds with decoder network G(:50g)
こ~ Ni,, o²), P. = (22 h1, h2, hy: 0g)
Note that both F.(:; (.) and G(::0g) are conditioned on Pigb Therefore, our generative model can be viewed as a conditional variational encoder (CVAE) [24] that learns the latent distribution of the 3D contact probability map Pe conditioned on the deep features of 3D color map Pigb- The conditional variational formulation takes observation Prph into account and can improve the generalization.
The encoders (igh(:; Orgh) and Fal:: share the same architecture and are instantiated with set abstraction layers from PointNet++ [23]. The Decoder G(: (g) instantiated with set upconv layers proposed in [25] to propagate features to existing up-sampled point locations in a learnable fashion.

---

Loss Function. Our generative model is trained to maximize the log- likelikeiiood of the generated contact probability map to be successful contact labels. Due to Jensen's inequality, the evidence lower bound objective (ELBO) in variational inference can be derived as
log P(Pc) 2 Pezuq(=lDe.Da)) log P(Pe|z, Pigb) Dx Pe: Prgb 1)) = --C (4)
The total loss C = Lrecom + Listent consists of two parts. Reconstruction loss Lrccom (==p(=lp log P(polz. Prgb)) = -Ez-g(=)p.) log P(pelz) = H (Pe. Pe) the cross entropy between Pe and Po. Latent loss Liarent = (NL(N | 1)) is the KL Divergence of two normal distributions. The goal of the training is to minimize the total loss C. The parameters in the three networks, Ogb.. lg and lg. are trained end-to-end to minimize L.
3.3 Plamning and Control Actions
Given the predicted contact points on object part i, the 6D pose of the end-effector of robot can be obtained by solving an inverse kinematics problem. Path planning algorithm such as RRT [26] is used to find an operational space trajectory from initial pose to desired grasping pose. If a path not found, the framework will iteratively sample contact point distributions from the generative model multiple times. Then we use Operational Space Controller [27] to move the gripper along the trajectory and finally complete the grasping action.
After a successful grasp, the next goal is to apply torques on robot arms and gripper to move the object part to the desired pose. To keep the grasping always valid, there should be no relative motion between the gripper and the object part. We propose Object-centric Control for Articuated Objects (OCAO) formulation. It computes the desired change of the 6D poses of robot end effectors given the desired change of the 6D pose of one of the object rigid parts and the changes of all joint angles. Operational Space Control (OSC) [27] will then be used to execute the desired pose change of the robot end effectors. For more technical details on the mathematical formulation of OCAO, please refer the supplementary material.
3.4 Additional Implementation Details
Part-level Pose Estimation. To ensure the object contact point is present in the 3D point cloud, we merge the colored object mesh point clouds and the scene point cloud combined point cloud. The 6D pose of each object part is estimated using DenseFusion [28]. an RGBD-based method for 6l object pose estimation. DenseFusion is trained separately using the simulation synthetic data. Since adding object mesh point cloud to the scene point cloud causes unnatural point density, we use farthest point sampling (FPS) on the merged scene point cloud to ensure uniform point density.
Automatic Label Generation from Exploration. Training the variational generative model re- quires a large amount of data. To collect sufficient data for training, we propose to automatically generate labels of contact point combinations using random exploration and interaction with the simulation environment. Expert-defined affordance areas of grasping, for example the handle of the plier, are provided to ensure realistic grasping and narrow down the exploration range. We provide initial human labeled demonstrations of possibly feasible contact point combinations, for example a pair of points at the opposite sides of the plier handle, to speed up exploration efficiency. We then use random exploration on the contact points within the affordance region and verify the tact point combinations through interaction in the simulation environment. Successful contact point combinations are saved in pool used in training. For more details on the algorithm of random exploration, please refer to the supplementary material.
Experiments
We design experiments to investigate the following hypotheses: 1) given training samples of enough variability, the contact point generative model can effectively learn the underlying distribu- tion; 2) contact point labels can be automatically generated through interaction with the environ- ment; 3) given sampled and generated contact points, after path planning and control, the robot is able to complete the manipulation task.
In this section, we present the experiments on manipulating two-part articulated objects from Part- Net [4] with two robot arms. To show that our framework can be applied to arbitrary number of
5

---

(a) 100144
(b) 100182
(c) 102285
(d) 10213
(e) 10356
(f) 11778
Figure 3: The six articulated objects from PartNet [4] used in our experiments. We illustrate their IDs in the dataset. The objects have two articulated parts connected by a revolute joint. The expert-defined affordance area the handles for the three pliers and the non-keyboard base area for the three laptops.
articulated object parts and robot arms, we further conduct experiments on manipulating a three- part articulated object with three robot arms. Please refer to the supplementary material for the details on the three-arm experiments.
4.1 Experiment Setup
Task and Environment.
We conduct the experiments in an environment constructed with MuJoCo [3] simulation engine. The environment consists of two robots facing a table in parallel. The artic- ulated object is placed on the table surface with random initial positions and joint angles. The goal of the manipulation is to grasp the object parts and move them reach the desired pose and joint angle configuration within a certain error threshold.
Objects and Robots. We evaluate our approach on six daily articulated objects from PartNet [4] in the experiments. The objects include three pliers and three laptops illustrated in Figure 3. For the robots in the experiment, parallel-jaw Rethink and Panda grippers. Both robots and grippers are instantiated by high-fidelity models provided in robosuite [29]. For the friction between robot gripper and the object, we assume a friction coefficient of 0.65 and assume pyramidal friction cone.
Sawyer and Panda which are equipped with two-finger
Baseline. There related previous work multi-arm articulated objects that our V-MAO with. So we developed baseline approach denoted as "Top-1 Point" that predicts a deterministic per-point contact probability map on the point cloud using encoder-decoder archi- tecture and selects the best contact point for each gripper finger within the affordance region. To fairly compare, the baseline uses the same encoder and decoder architectures and is trained on the same interaction data as V-MAO. After contact point selection, the subsequent planning and control steps are also the same for the baseline.
compare
4.2 Quantitative Evaluation
We explore two types of grasping strategy, sequential grasp and parallel grasp. In sequential grasp, object parts grasped and moved in sequential fashion. In i-th step, robot gripper i grasps the i-th object part while leaving parts + 1 through N uncontrolled. We iterate the above step until all parts grasped. In parallel grasp all object parts grasped simultaneously. Then the robots move the object parts to the goal locations together.
We report the success rate of manipulation in Table 1. Our V-MAO can achieve more than 80% suc- cess rate with Sawyer robot and more than 70% success rate with Panda robot on all objects, which shows V-MAO can effectively learn the contact distribution and complete the task. Besides, the Panda robot, sequential grasping generally achieves a much better success rate than parallel grasp- ing. This means determining grasping for other robots later may be a better option than determining both robots' grasping at the beginning in certain cases.
We notice that when using the Panda robot, the performance of both V-MAO and the baseline de- crease significantly. The reason that the Panda robot has large end effector size. Therefore when manipulating small objects like pliers, in many cases the space is not enough to contain both grip- pers due to robot collision, while Sawyer robot's end effector much smaller. The deterministic prediction baseline achieves a larger success rate on laptops. A possible explanation is that there

---

robot
method
grasp order
100144
100182
102285
10213
10356
11778
parallel
90.0
86.0
96.0
88.0
90.0
Top-1 Point
sequential
82.0
78.0
88.0
92.0
84.0
86.0
Sawyer
parallel
94.0
90.0
90.0
96.0
90.0
94.0
V-MAO (Ours)
sequential
92.0
86.0
92.0
96.0
88.0
90.0
parallel
50.0
20.0
48.0
90.0
92.0
88.0
Top-1 Point
sequential
62.0
70.0
78.0
82.0
70.0
Panda
parallel
52.0
36.0
64.0
90.0
84.0
88.0
V-MAO (Ours)
sequential
72.0
76.0
78.0
88.0
86.0
86.0
Table 1: Success rate of Manipulation. The baseline we compare V-MAO with is selecting the point with the maximum contact probability from deterministic prediction. We report the results averaged from 50 runs.
(a)
(b) (c) -0-11 1.0
(d) - 0.5
(e) = 0.5
(l) 1.0
Figure 4: Visualization of sampled and generated 3D probability maps. From left to right in each row: (a) the current manipulation state; (b) converted colored 3D point cloud at the current state; (f)-(f) generated probability map of the next grasp given specific latent code z, where red and yellow denote the finger 1 and of the first gripper in the first grasping step, and green and blue denote the finger and of the second gripper the second grasping step. Zoom for better a view of the figures.
less variation in grasping laptops compared pliers, for example the first grasp of the laptop is almost always grasping the base horizontally. Thus a deterministic model can be more stable in simpler tasks. On the other hand, in environments with larger variability, generative model can have advantage.
4.3 Qualitative Results
In Figure 4, we visualize the probability map on the 3D point clouds sampled from the generative model given different latent codes z. As expected, the maximum value probability value falls on the object points. The points that have the highest probability for the respective gripper also form valid grasps. Given different latent codes 2, the generated contact points can cover a wide range of feasible contact point combinations. We also notice that there some object regions that are
7

---

front view
side view
front view
side view
Figure 5: Visualization of two sequential manipulations using two robot arms. Rows I and
2 illustrate the manipulation of pair of pliers (100144) using two Sawyer robots.
Row 3 and The object
4 illustrate the manipulation of a laptop computer (10213) using two Panda robots. configuration goal of manipulation is marked by the green trassprrnnp point cloud.
less covered by the generative model during sampling. One possible reason is that grasping in these regions can introduce collisions in the current or the next steps and have lower probability of being sampled.
In Figure 5. visualize the executed sequential manipulation using the contact points sampled from the generative model. As we can see, the robot gripper can successfully reach the contact points after inverse kinematics and path planning. When both grippers are grasping the object, the sampled and generated contact points combination for both robots can also avoid collision between robots. After grasping, both robots can reliably control the object reach the goal configurations marked by green point cloud.
5 Conclusion
In this paper, we propose V-MAO, a framework for learning multi-arm manipulation articulated objects. Our framework includes variational generative model that learns contact point distribution over object rigid parts for each robot arm. We developed a mechanism for automatic contact label generation from robot interaction with the environment. We deploy framework in customized MuloCo simulation environment and demonstrate that our framework achieves high success rate on six different objects and two different robots. V-MAO achieves 80% success rate with Sawyer robot and over 70% success rate with Panda robot. Experiment results also show that the proposed variational generative model can effectively learn the distribution of successful contacts. As future work, we will investigate articulated object manipulation modeling using other sensor modalities such stereo RGB images [30. 31] and dynamic point clouds [25, 32], or other pose estimation methods such as [33, 34].

---

Acknowledgments
This work is funded in part by JST AIP Acceleration, Grant Number JPMJCR20U1, Japan.
References
[1] H. J. Xu, and S. Song. Learning a decentralized multi-arm motion planner. In CoRL, 2020.
[2] V.N. Hartmann, A. Orthey, D. Driess, O. Oguz, and M. Toussaint. Long-horizon multi-robot rearrangement planning for construction assembly. arXiv preprint arXiv:2106.02489, 2021.
[3] E. Todorov, T. Erez, and Y. Tassa. Mujoco: IROS, 2012.
A physics engine for model-based control. In
[4] K. Mo, S. Zhu, A. X. Chang, L. Yi, S. Tripathi, J. Guibas, and H. Su. Partnet: large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In CVPR, 2019.
[5] D. P. Kingma and M. Welling. 14X1v:1312.611,,000 2013.
Auto-encoding variational bayes.
arXiv preprint
[6] I. J. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. In NIPS, 2014.
[7] A. Wang, T. Kurutach, K. Liu, Abbeel, and A. Tamar. Learning robotic manipulation through visual planning and acting. RSS, 2019.
[8] T. Kurutach, A. Tamar, G. Yang, S. Russell, and P. Abbeel. Learning plannable representations with causal infogan. arXiv preprint arXiv:1807.0933 09341, 2018.
[9] D. Morrison, P. Corke, and J. Leitner. Closing the loop for robotic grasping: real-time, generative grasp synthesis approach. In RSS, 2018.
[10] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In ICCV, 2019.
[11] J. Lundell, E. Corona, T. N. Le, F. Verdoja, P. Weinzaepfel, G. Rogez, F. Moreno-Noguer, and V. Kyrki. Multi-fingan: Generative coarse-to-fine sampling of multi- finger grasps. In ICRA, 2021.
[12] M. S. Kopicki, D. Belter, and J. L. Wyatt. Learning better generative models for dexterous, single-view grasping of novel objects. IJRR, 2019.
[13] J. Lundell, F. Verdoja, and V. Kyrki. Ddge: Generative deep dexterous grasping in clutter. arXiv preprint arXiv:2103. 04783, 2021.
[14] X. Chen, R. Chen, Sui, Z. Ye, Y. Liu, Bahar, and O. C. Jenkins. Grip: Generative robust inference and perception for semantic robot manipulation in adversarial environments. In IROS, 2019.
[15] H. Ha, S. Agrawal, and S. Song. Fit2form: 3d generative model for robot gripper form design. CoRL, 2020.
[16] D. Katz, Y. Pyuro, and O. Brock. Learning manipulate articulated objects in unstructured environments using a grounded relational representation. In RSS, 2008.
[17] D. Katz and O. Brock. Manipulating articulated objects with interactive perception. In ICRA, 2008.
[18] K. N. Kumar, I. Essa, S. Ha, and C. K. Liu. Estimating mass distribution of articulated objects using poe-preee manipulation. arXiv preprint 1rXiv:1977.0364 2019.
[19] V. Narayanan and M. Likhachev. Task-oriented planning for manipulating articulated mecha- nisms under model uncertainty. In ICRA, 2015.

---

[20] M. Mittal, D. Hoeller, F. Farshidian, M. Hutter, and A. Garg. Articulated object interaction in unknown scenes with whole-body mobile manipulation. arXiv preprint arXiv:2103. 10534, 2021.
[21] R. Shome and K. E. Bekris. Anytime multi-arm task and motion planning for pick-and-place of individual objects via handoffs. In International Symposium on Sulti-RRbot and Multi-Agent Systems (MRS), 2019.
[22] R. Shome, K. Solovey, J. Yu, K. Bekris, and D. Halperin. Fast, high-quality two-arm rearrange- ment in synchronous, monotone tabletop setups. IEEE Transactions on Automation Science and Engineering, 2021.
[23] C. R. Qi, L. Yi, H. Su, and J. Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in metric space. In NIPS, 2017.
[24] K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. NIPS, 2015.
[25] X. Liu, C. R. Qi, and L. Guibas. Flownet3d: Learning scene flow in 3d point clouds. In CVPR, 2019.
[26] S. M. LaValle al. Rapidly-exploring random trees: A new tool for path planning. 1998.
[27] O. Khatib. Inertial properties in robotic manipulation: An object-level framework. IJRR, 1995.
[28] C. Wang, D. Xu, Zhu, R. Martin-Martin C. Lu, L. Fei-Fei, and Savarese. Densefusion:
6d object pose estimation by iterative dense fusion. In CVPR, 2019.
[29] Y. Zhu, J. Wong, A. Mandlekar, and R. Martin-Martín. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. 12293, 2020.
[30] X. Liu, R. Jonschkowski, A. Angelova, and K. Konolige. Keypose: Multi-view 3d labeling and keypoint estimation for transparent objects. In CVPR, 2020.
[31] X. Liu, Iwase, and K. M. Kitani. Stereobj-Im: Large-scale stereo image dataset for 6d
object pose estimation. In ICCV, 2021.
[32] X. Liu, M. Yan, and J. Bohg. Meteornet: Deep learning on dynamic 3d point cloud sequences. ICCV, 2019.
[33] X. Liu, Iwase, and Kitani. Kdfnet: Learning keypoint distance field for object pose estimation. arXiv preprint arXiv:2109, 10127, 2021.
[34] S. Iwase, X. Liu, R. Khirodkar, R. Yokota, and K. M. Kitani. Repose: Real-time iterative rendering and refinement for 6d object pose estimation. arXiv preprint arXiv:2104.00633, 2021.
10


Reviews:
[{"review_id": "PFweKDP8xTc", "reviewer": "Reviewer_War1", "summary": "This paper aims to tackle the problem of multi-arm manipulation of articulated objects. The authors propose V-MAO, a framework that learns a variational generative model to predict the contact point distribution over object rigid parts for each robot arm to manipulate. They obtain the training signals via interactions with the simulation environment enabled by object-centric planning using the current model.\n\nThe authors deploy the framework in a customized MuJoCo simulation environment and demonstrate that it achieves a relatively high success rate on six different articulated objects with two rigid parts using two different robots. They have also shown that the generative model used in the framework allows it to outperform a deterministic baseline.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strength]\n\nThis paper targets an interesting yet challenging problem of manipulating articulated objects. The results in the simulation suggest that the model can manipulate several articulated objects and successfully achieve the desired configuration.\n\n\n[Weakness]\n\nWhile I like the direction this paper is going, I have concerns regarding its ability to deploy to the real world, the claims made in the paper, and the detailed design of the method. \n\nThis paper only shows experiments in simulation. Will the method generalize to the real world? Are there any missing components or potential problems when extending to the physical robots? The authors say that \"the training signal is obtained from interaction with the simulation environment.\" Can the model trained in simulation directly generalize to the real world, or do we have to obtain the label via real-world interactions? The authors may consider providing more details on how much and what quality of interaction data is needed.\n\nRelated to my previous point, the method seems to require labels on object parts and the grasp probability map. Obtaining these labels in simulation may be straightforward but not in the real world. Can models trained using these labels in simulation generalize to the real world, or do we have to manually label them directly on the real data, and how well can the learned model generalize?\n\nThe authors claim that \"V-MAO achieves over 80% success rate with Sawyer robot and over 70% success rate with Panda robot,\" and they have provided some videos of the successful trials, which is profoundly helpful. However, no failure examples are provided, making it hard to fully understand the performance of the method. The authors may consider showing the failure cases, analyzing them, and summarizing the common failure scenarios.\n\nRelated to my previous point, the authors may also consider providing failure cases for the deterministic baseline and include more explanations of why it is worse than V-MAO. The generative model is learned by giving the deterministic labels. Are there any additional supervisions or mechanisms that allow the proposed generative model outperforms the deterministic baselines?\n\nThe authors claim multiple times in the paper that the method is developed to handle objects with N rigid parts, where N >= 2. However, the authors only provide experimental results on objects consist of two rigid parts. Without concrete experimental results on objects with N > 2, I suggest the authors adjust their claims to set the right expectation for the readers.\n\nThe authors describe in Section 3.4 that the grasping areas are hand-specified, and they also need to provide initial demonstrations of feasible contact point combinations. These are important design choices for making the pipeline work. The authors may consider providing more details like how many demonstrations are needed for each object? What if there are multiple locations that could lead to successful grasps? Can it generalize to objects with similar but different geometries, e.g., objects in the same category?\n\nHow did the authors plan the grasps when given the grasping locations? Is it based on the local geometry of the object or expert demonstrations?\n\nThis paper contains an extensive amount of typos and grammatical errors. Detailed proofread would be necessary. Just to name a few:\nLine 2: c collaboratively --> collaboratively;\nLine 31: uncertainly --> uncertainty;\nLine 42: desire poses --> desired poses;\nFigure 2 caption: Each --> , each;\nLine 82: a object-centric --> an object-centric;\nLine 217: V-Mao --> V-MAO\n\n============================\n\nPost-rebuttal\n\nI have read the reviews from other reviewers and the rebuttal from the authors. I appreciate the authors' efforts in providing the additional experimental results and the revision to the paper.\n\nThe authors' response has addressed most of my concerns, and I'm now supportive of this paper to present at the conference. I have changed my score from Weak Reject to Weak Accept.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While I like the direction this paper is going, I have concerns regarding its ability to deploy to the real world, the claims made in the paper, and the detailed design of the method. I hope the authors can address my issues, and I'm willing to increase the score to reflect the changes.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "I would love to see results on the real robots, but I understand that real-world deployment may be challenging due to the pandemic. Still, I hope the authors can address the issues raised in the main review, after which I would be willing to increase my score."}}, {"review_id": "BDXvOHzvXz", "reviewer": "Reviewer_Fyb6", "summary": "This paper presents a generative approach to learn manipulation of articulated objects for multiple robot arms. The generative model learns a distribution of contact points on 3D point clouds. The approach is validated in simulated Mujoco environment.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The approach seems very effective for multi-arm manipulation on articulated objects under two different grasping conditions. \n\nWeakness:\n- Section 2 shows a list of existing works on relevant topics, but it is confusing about the novelty of the proposed approach from the writing. Is the novelty on the side of learning a generative model on articulated objects? Or a generative model for grasping points? My understanding is that the contribution lies in the former one. In that case, what is the relationship between the proposed approach and some related works which also use generative approach on articulated objects (such as [1]) ?\n- Before authors can address the concern between the current approach and existing works more extensively, the work seems only incremental to me, especially only evaluated in the simulation instead of on a real robot.\n- Also, the formulation states that this is for multi-arm case where N>=2. However, all experiments are only done on two robot arms, without any quantitative or qualitative experiments to show three-arm cases. In that case, I think stating N>=2 is not convincing that the proposed approach can extend to N=3 case. I suggest authors either add more experiments or make more upfront formulation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I rate clarity of presentation fair because at the current stage, it's hard to evaluate if the contribution is enough of the work. It is more like an incremental work, combining some generative approach and apply it in a bi-manual case. Overall the execution of the paper is good, but it fails to make the contribution clearly and also have some misleading statements such as the approach applies to N>=2 cases where there are no experiments to demonstration N>2 cases.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- Index of gk or gn in the problem formulation\n- In Sec 3.3, I am confused by “another Operational Space Controller”. Does it mean the controller has a different parameter from the one for grasping? Or are they the same controller?\n- The data collection is briefly mentioned, but I miss the details of how many training samples required for each object. Also, how the point clouds of the first robot arm is processed during data collection?\n- I am not clear about the part of superimposing the object mesh point clouds. What does this step exactly mean?\n- Some grammar issues such as, line 81 ‘object geometric’,  line 208 “iterative the above step” \n"}}, {"review_id": "2O9DZ9Nm4Gs", "reviewer": "Reviewer_E39s", "summary": "This paper presents V-MAO, an architecture to learn multi-arm manipulation of articulated objects. It computes feature representations for 3D point clouds and a probability distribution over a latent variable, representing the contact points. Inverse Kinematics is used to compute the 6D pose of the robot end-effector given the predicted contact points on the object. RRT is used to compute collision-free paths. An Operational Space Controller is used to move the gripper. The 3D point cloud might contain only an object, with multiple parts, or also points corresponding to one of the grippers, which is important to achieve collision-free multi-arm manipulation.\n\nThe training data is obtained by using expert-defined affordance areas for grasping plus exploration by the robot.\n\nThe learning architecture is trained to maximize the log-likelihood of the generated contact points to lead to successful grasps.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This work tackles a challenging manipulation problem and produces convincing results in simulation.\n\nWeaknesses:\n- The presentation could be made more clear.\n- The challenges on the way of applying this  method to real robots should be discussed.\n\nIdea for future work:\n- Consider multimodal distributions for the latent variable to be able to grasp at multiple disjoint positions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper tackles a challenging problem in a technically sound manner and achieves very good results, as shown in the submitted video.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Can the authors discuss the main challenges to apply the proposed method to real robots?\n\nSometimes, the index \"i\" is used to denote several elements at the same time, e.g. robot i, object part i. Consider using different indexes to denote different elements.\n\nIn Section 3.1, why is P_S^(k) introduced. It does not seem to be used anywhere in the paper.\n\nIs the latent variable just a scalar? Why not a vector? Wouldn't it be better to model the distribution over z by a multivariate Gaussian with full covariance matrix?\n\nCan the authors make the subsection \"Part-level Pose Estimation\" more clear? What's the issue with superimposing object mesh points? How does farthest point sampling (FPS) work?\n\nA visualization of the baseline architecture \"Top-1 Point\" in the paper or in the supplemental material would be helpful.\n\nSection 4.2: \"Besides, sequential grasping generally achieves a better success rate than parallel grasping. This means determining grasping for other robots later may be a better option than determining both robots’ grasping at the beginning.\" --> Actually, for the Sawyer robot, parallel was better (Table 1).\n\nTypos:\n- Abstract: \" It is challenging to enable multiple robot arms to c collaboratively complete manipulation tasks on articulated objects\" --> collaboratively\n- 1. Introduction: \"In this paper, we propose a framework name V-MAO for learning manipulation of articulated objects\" --> named\n- Figure 1: \"Grasp & Manipuation\" --> Grasp & Manipulation\n- Section 2: \"Other works has proposed to use generative model to learn more complex grasping\" --> have proposed\n- Section 2, Articulated Object Manipulation: \"Katz et al. [15] proposed a relational state representation of of object articulation for learning manipulation skill\" --> \"of\" is repeated\n- Section 2, Articulated Object Manipulation: \"For example, from interactive perception, Katz et al. [16] learns kinematic model of an unknown articulated object through interactive perception.\" --> \"interactive perception\" twice\n- Section 2, Multi-Arm in Robotic Manipulation: \"we focus on the learning the contact point distribution on object parts\" --> focus on learning\n- Section 3.1: \"we use conditional probability to decompose P and assumes previous point clouds have no effect on the current grasping\ndistribution\" --> assume\n- Section 3.2: \"only the pixel value distribution need to be modeled\" --> needs\n- Section 4: \"We design our experiments to investigate the following hypothesis\" --> hypotheses (plural)\n- Figure 3: \" We illustrates their ID in the dataset\" --> illustrate\n- Section 4.2: \"We iterative the above step until all parts are grasped\" --> iterate over the above step...\n- Table 1: \"Saywer\" --> Sawyer"}}]

Author rebuttal:

Meta-review: Strengths:

- All of the reviewers felt the general direction of the work was interesting. There was a sense that the variational approach was interesting here.

- The reviewers felt that the simulated experiments effectively demonstrated that the approach works well in simulation.

Weaknesses:

- A key concern from the reviewers was that the approach was evaluated primarily in simulation. While there are real robot videos, there are little/no quantitative results.

- The AC had a question about the high level motivation of this paper. The approach assumes we are given full geometric object models. And, the method requires accurate part-level pose estimation (section 3.4). So, why not plan grasp points using traditional methods instead of the variational approach?

- The variational approach here seems closely related to the approach in “6-DOF GraspNet: Variational Grasp Generation for Object Manipulation” by Arsalan Mousavian, Clemens Eppner, Dieter Fox. However, in that work, the authors augment the variational sampling step w/ subsequent classification and grasp adjustment in order to get accurate grasp placement. Why was that not also necessary here?

- There was a question about where the ground truth labels came from for training the model. Were these hand-labeled? How large was the dataset / how many human demonstrations were provided?

- The baseline method, “Top-1 Point”, was not described very clearly in the main text and there is no other baseline.

- There was a concern that since the method is only evaluated for two robotic arms, it is unfair to assert the method works for larger numbers of arms.

Post-rebuttal:

We appreciate the new three-arm experiment strengthens the paper and the revisions for clarity.


Decision: Accept (Poster)