Composable Part-Based Manipulation
Authors: Weiyu Liu
Jiayuan Mao
Joy Hsu
Tucker Hermans
Animesh Garg
Jiajun Wu
Venue: CoRL 2023 Poster (2023)
Keywords: Manipulation
Part Decomposition
Diffusion Model

Abstract: In this paper, we propose composable part-based manipulation (CPM), a novel approach that leverages object-part decomposition and part-part correspondences to improve learning and generalization of robotic manipulation skills. By considering the functional correspondences between object parts, we conceptualize functional actions, such as pouring and constrained placing, as combinations of different correspondence constraints. CPM comprises a collection of composable diffusion models, where each model captures a different inter-object correspondence. These diffusion models can generate parameters for manipulation skills based on the specific object parts. Leveraging part-based correspondences coupled with the task decomposition into distinct constraints enables strong generalization to novel objects and object categories. We validate our approach in both simulated and real-world scenarios, demonstrating its effectiveness in achieving robust and generalized manipulation capabilities.

Composable Part-Based Manipulation
Weiyu Liu', Jiiyuan Mao", Joy Hsu', Tucker Hermans Animesh Garg Jiajun Wu Stanford MIT .NVIDIA *University of Utah *Georgia Tech
Abstract: In this paper, we propose composable part-based manipulation (CPM), novel approach that leverages object-part decomposition and part-part correspon- dences to improve learning and generalization of robotic manipulation skills. By considering the functional correspondences between object parts, we conceptualize functional actions, such pouring and constrained placing, as combinations of different correspondence constraints. CPM comprises a collection of composable diffusion models, where each model captures different inter-object correspon- dence. These diffusion models can generate parameters for manipulation skills based the specific object parts. Leveraging part-based correspondences coupled with the task decomposition into distinct constraints enables strong generalization to novel objects and object categories. We validate approach in both simulated and real-world scenarios, demonstrating its effectiveness in achieving robust and generalized manipulation capabilities. For videos and additional results, see our website: https//epmoor12223.ghhue
Keywords:
Manipulation, Part Decomposition, Diffusion Model
Train: Pour from glass, and bowl to bowl in simulation
Test: Pour from mug to bowl in real world
rilm
handle
body
Pour
<olign, rim>
<facing-up. handle, body>
<tilt, body, body>
Figure CPM composes past-based diffusion models predict target object poses directly from point clouds. this example, we show that the "pouring" action is decomposed into three part-based correspondences, which generalize manipulation across object categories, and from simulation to the real world
Introduction
Compositionality provides appealing benefits in robotic manipulation, it enables efficient learning, reasoning, and planning. Prior works have extensively studied the decomposition of scenes into objects and their relationships [1, 2, 3], as well as the division of long-horizan plans into primitive skills [3, 4], order to navigate complex environments and devise long-horizon plans. this paper, we present a different view of compositionality by considering object-part decomposition based on functionality (c.g., rim, handle, body), and leverage such decomposition to improve the learning of geometric and physical relationships for robot manipulation.
In the context of language descriptions of objects, part names not only describe the geometric shapes of the parts but also capture their functional affordances. For instance, as depicted in Figure 1, for the action of "pouring", the rims define the boundary for alignment between the objects, the body pouring vessel should be tilted for the action, and its handle provides constraint on the direction the object should face when pouring. Leveraging this knowledge of part affordances, family of functional actions, such as pouring and constrained placing, can be conceptualized as combination of functional correspondences between object parts. Modeling actions using such
posit that
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

decomposition yields two important generalizations. First, it enables action generalization to novel instances from the same object category. Second and more importantly, it facilitates generalization to unseen object categories. For example, after learning part affordances for the "pouring" action, our robot trained on "pour from bowls" and "..pans" can generalize to "pour from mugs", with no additional training necessary for manipulation with the new object category.
Motivated by these insights, we present the composable part-based manipulation (CPM). CPM comprises a collection of diffusion models, where each model captures the correspondence between parts of different objects. These conditional diffusion models take the geometry of the object parts as input and generate parameters for manipulation skills, such as the starting and ending poses of a bowl during the pouring action. Specifically, each model outputs distribution of feasible trajectories that satisfy a particular correspondence. After learning a collection of composable diffusion models, represent actions as combinations of part-part correspondences. During inference, we leverage the composition of primitive diffusion models to sample trajectories that adhere to all the part correspondences. This approach improves generalization to novel object categories over models that do not reason about both parts and composable correspondence constraints.
In summary, this paper makes two key contributions. First, we propose composable part-based manip- ulation, which models manipulation actions as composition of part-part correspondences between objects. Second, we develop diffusion models trained to capture primitive functional correspondences that can be flexibly recombined during inference. CPM achieves strong generalization across various dimensions, including novel object instances and object categories. We validate the efficacy of CPM on both PyBullet-basee simulations and real-robot experiments.
2 Related Work
Object representations for manipulation. Prior works use segmentations of common object parts (e.g.. blades, lids, and handles) for manipulating articulated objects [5, 6, 7, 8] well for transfer to novel objects [9, 10]. A common approach that has been shown effective across different manipulation domains [11, 12, 13] first predicts which part of an object the robot should focus on (e.g., the handle), and then predicts an action relative to the part. Closely related visual affordance detection [14, 15, 16], which segments objects into different functional regions, such as graspable parts and support surfaces of objects. These functional regions can be shared by more distinct objects, and can be useful for generalizing task-oriented grasping between object categories [17, 18]. Keypoints are another representation that shows robustness to large intra-category shape variation and topology changes [19]. Each keypoint set can provide essential pose information, that lacks in previous segmentation approaches, to support tasks such as hanging mugs on pegs by their handles. The initial supervised approach [19] has been extended to methods that discover keypoints from interactions [20, 21] and from unlabeled videos [22]. Recently, implicit object representations have been used to provide correspondence between any point within the same object category generalizing across 6- G-DoF pose changes [23, 24, 25]. Large pretrained vision models also support the development of object representations: recent works leverage these models to significantly reduce domainn-spccinn training data, showing strong results for open- ppen-vocabllary part segmentation [26], few-shot affordance segmentation [27], and one-shot pose estimation on any novel object from the same category [28]. Despite this huge progress, we still lack object representations that support strong generalization of manipulation to new object categories. We focus on tackling this problem.
Learning interactions of objects. Works in robotics have established the importance of modeling interactions of objects. Recent approaches directly work on 3D observations, without relying on known object models. Learning spatial relations between objects enables the picking and placing of objects at specific locations [1, 29, 30, 2, 31], such as placing an object in the middle drawer, stacking objects, and setting the table. These relations can be extended to represent the logical state of the world to support planning for long-horizon tasks [3, 32, 33). Other works focus on learning lower-level interactions between objects, such as placing an object stably on a messy tabletop and pushing object using tool [34, 35]. For example, -20-afford [34] correlates feature maps extracted from two objects using point convolution and outputs point-wise interaction heatmap.

---

(a) System Overview
"Pour"
(c) Point-Clous Diffusion Transformer
Pg
(C)
AF
S.1
S...
E
Soone
Segmented Parts
Sampled Targe Poses
Farcution
Diffusion Transformer
[b) Composable Part- Based Diffusion Models
Point Cloud Transformrr
Poss Enooder
Σ
(e) P
(e)
E6, align
E6, f6.facing up
Eg,eslt
P
XPA XPF.
(2)
T
PSAA
Figure 2: (a) Given task, the partial point clouds of the anchor and function objects, and their parts extracted from learned segmentation model Po. we sample a sequence of transformations from a learned distribution ps to parameterize the function object's 's trajectory. (b) CPM can be generalized to novel object categories because it decomposes each action to a collection of functional correspondences between object parts. To sample the larget transformations that satisfy all functional correspondences, CPM combines the noise predictions from collection of primitive diffusion models at inference time. (c) Each primitive diffusion model learns target posc distribution that satisfies a particular part-pat part correspondence, based on the point clouds of the object parts.
Functionals defined on top of object- wise signed distance functions can also represent constraints on interactions between objects such as contact and containment [36]. Plow-based methods can also learn static relations between objects [37] as well as tool use [38], directly from point clouds. A main difference between our work and these methods is that we bridge the modeling of interactions and object representations through object-pau -art decomposition and learned part-part correspondences, and enjoy empirically validated improvement in generalization.
Composable difflusion models. A set of recent works have investigated the potential of diffusion models in robotics [39, 40, 41, 42, 43, 44, 45, 46, 2, 47]. Research demonstrates that diffusion models generate multimodal distributions over actions [41] and can handle spatial ambiguities in symmetric objects [2]. In image domains, prior work has shown connection between conditional diffusion models and energy-based models, and proposed techniques to generate images by combining diffusion noises for different language conditions [48]. Recent work provides a more principled way to sample from individually trained models using MCMC [49]. Another approach combines diffusion models by using additional trained adapters for generating faces [50]. CPM combines both lines of work to propose composable diffusion models for robotic manipulation. In doing so must address two challenges of adapting diffusion models to (I) output poses instead of pixels and (2) combine actions in different part frames, while retaining generalization to different distributions.
3 Composable Part-Based Manipulation
In this work, our goal is to model functional actions involving an anchor object A that remains static and function object F that is being actively manipulated. Shown in Fig. 2 (a), given task M and the partial point clouds of two objects X and X in the world frame ( (W), we want to predict sequence of SE(3) transformations, i.e., Tw = [Tw.1:-.. Tw ,N J. which parameterized a trajectory of the function object F in the world frame in order to achieve the desired interaction with the anchor object (e.g., pouring). Throughout the paper, we choose N = 2; i.c., we predict the starting pose and the ending poses of the object motion. Then, we use SE(3) interpolation between the two poses generate the continuous motion trajectory. We define that the object frames of (A} and (F) centered the centroids of the respective point clouds X A and X and have the same orientation as the world frame Each transformation Tw in the world frame can thus be computed by the relative pose between the two objects TAF as Tw TWATTAR ATAP(TTe (TwP) (iw))- A key challenge we aim to address is generalizing the functional actions from training objects to unseen object instances and, more importantly, novel object categories a robot may have never encountered during training.
"The transformation from {W} to an object frame can be computed given this definition. For example, Twr (RwF, twF), where Rw is to identity matrix and twr is set the centroid of XF.
3

---

3.1 Action as Part-Based F Functional Correspondences
Composable part-hased manipulation (CPM) models each action M as composition of functional correspondences between object parts. We formalize the symbolic representation of each correspon- dence C' CM (S,, PA. a) where Cm is the set of correspondences for M, S. is spatial relation, PA.j and PF are two parts of the anchor and the functional objects, respectively. Consider the example of pouring from a mug to a bowl, as depicted in Fig. 1. This "pour" action contains the following three correspondences: (align, rim(mug), rim()owl)), (filt, body(mug), body(bowl)), and (facing-up up, handle (mug), body(bowl)..
The task of predicting robot motion can be cast as the task of finding robot trajectory that simultane- ously satisfies all the part- based functional correspondences. Instead of manually specifying these constraints given object point clouds and their poses, we propose to learn neural network do to recognize the funcctiona parts of objects based on their point clouds and another learned generative model Po to parameterize a distribution of T. Using Is. we can extract point clouds for given part, for example (x(XF. PF = Xpr Learning to recognize functional parts can be treated as predicting per-point part segmentation problem and have been studied extensively in prior work [14, 15, 16, 27, 51J. Therefore, we focus on the second part which enables the robot to learn manipulation trajectories of objects, based on the recognized parts.
3.2 Generative Modeling of Functional Correspondences with Diffusion Models
For each functional correspondence tuple (S,, PA.j, PF,s), we learn generative distribution Pe.s, XPr..) Here Tp₁, denotes the relative transformations Tr.., We use a point-cloud conditioned diffusion model to parameterize this distribution. In particular, each primitive diffusion denoise model co.s, takes in the current diffusion time step f, two part point clouds X PA.; and XPF. and the noisy transformations Tp as input, and predicts the noise over T As illustrated in Fig. 2 (c), the model based on a transformer encoder. First, we encode point clouds for the parts separately using a point cloud transformer [52]. Then we encode each transformation using trained MLP. We input the point cloud and transformation encodings, together with the diffusion time step to the transformer encoder. The output of the transformer encoder is the predicted noise over the transformations TP,,. We provide details for the architecture in Appendix A.
During training, we optimize the following loss for randomly sampled diffusion time step t and random Gaussian noise e sampled from multivariate Gaussian distribution:
MMSE = ||
c0.s; (Vi - (0) VBc X X X
1) 11:
where Tppy is the target transformations predict and is the diffusion noise schedule [53]. The added noise and the predicted noise are both in the tangent space of SE(3). We build on the technique introduced for the SE(3) Denoising Score Matching (DSM) model [40], but use Denoising Diffusion Probabilistic Model (DDPM) [53] for more stable training. In practice, we first compute the exponential map of the transformations and then apply the noise. This can be viewed predicting the score function for exponential energy function of SE(3) poses.
3.3 Inference-Time Composition of Diffusion Models
One of the key features of diffusion models is their compositionality. That is, suppose we have set of diffusion models, each trained for one specific type of functional correspondences, we can combine their predicted noises during inference time to generate a trajectory that adheres to all functional correspondences, as illustrated in Fig. (b). Since each diffusion model implicitly parameterizes an energy-based model: pa,s, (TI-) x exp(-Eee -Ea, s, (TI-)) through its noise prediction [48, 49], sampling from the composition of the diffusion models corresponds sampling from the "intersection" of distributions for the individual functional correspondences, or formally, from IIcec (T
SSiilarr to the definition of the object frame. the part frames (PA.) and (Pr..} are centered at the centroids of the respective point clouds XPr, and X PF.s and have the same orientation the world frame.

---

Pour from mug to bowl
Pour from bowl to mug
Place knife safely in mug
Figure 3: We generate task demonstrations using the PartNet and ShapeNetSem datasets for the "pouring" and "safe placing" tasks. We create demonstrations for a variety of function and anchor object combinations.
In particular, during inference time, starting from T(P) randomly sampled from standard Gaussian distributions, given the set of constraints CM, iteratively update the pose prediction by:
1 ae
T
1 at Σ co.s. (fro.pert (TAP) XPAS Xpeast) + ore,
V di CECи
where T' is the number of diffusion steps, a = =1 -1--, 3e is the denoise schedule, ã₁ = II ag the cumulated denoise schedule, at fixed sampling-time noise schedule, and is randomly sampled Gaussian noise. The differentiable operation feo.part takes T(t) and transforms it to the part frame Pys by (TAPA,)-'TAP TFPE. ... for which each individual diffusion model trained on.
4 Data Collection
We demonstrate CPM on the "pouring" and "safe placing" tasks. These two tasks require different funetional affordances. The pouring action pours from an anchor object target object, and requires alignment of rims, collision avoidance of the handle and the container body, and body tilt. The safe-placing action places sharp function object into an anchor object, and requires head containment for safety, tip touching bottom, and a body-body placement constraint. To validate our approach, we collect 4522 successful demonstrations for pouring and 2836 successful demonstrations for safe placing. generate the demonstrations, we first source 13 categories of 3D objects from PartNet [54] and the subset of ShapeNetSem [55] objects categorized in the Acronym dataset [56]. We then extract aligned parts either from segmentations and category-level canonical poses in PartNet from manually labeled 3D keypoints for ShapeNetSem objects. We procedurally generate parameters of the actions from the aligned parts (as illustrated in Fig. 3), simulate the interactions by tracing the trajectories defined by the parameters, and render RGB-D images using multiple cameres set in the simulator. Details of the dataset are presented in Appendix C.
5 Experiments
The subsequent section will showease the performance CPM in comparison to baselines and other variants of our method in simulation. In particular, we evaluate in two important generalization settings: generalization novel object instances from seen object categories, and 2) generalization to object instances from unseen object categories. We then discuss the deployment of CPM trained in simulation real robot.
5.1 Experimental Setup
We evaluate all methods the PyBullet physics simulator [57]. To isolate the problem of predicting target transformations from other components of the system (e.g., grasp sampling and motion planning), we actuate the center of mass of the function object F. We report average task completion scores from 1500 trials indicating failure (0) and success (100), with credits assigned for partial completion. The score is computed based on model-basee classifiers designed for each task. To test

---

Table 1: CPM demonstrates strong generalization to novel instances of objects within seen categories.
Model
Safe Placing Pouring
Transformer-BC.
19.21
37.11
TAX-Pose
21.71
76.97
PC-DDPM
75.83
51,55
Part-Aware PC-DDPM
75.28
42.68
CPM (ours)
80.00
70.99
Table 2: CPM demonstrates strong generalization to function objects from unseen object categories.
Pouring
Safe Placing
Model
Bowl
Glass
Mug
Pan
Fork
Pen
Scissors
Transformer-Bi
10.23
20.91
6.06
32.04
26.15
31.93
26.44
TAX-Pose
23.32
3.82
8.64
46.14
50.90
67.60
36.80
PC-DDPM
63.02
75.95
71.39
64.39
40.60
46.63
32.34
Part-Aware PC-DDPM
58.98
72.11
67.11
66.17
39.76
48.04
28.15 15
CPM (ours)
79.32
81.44
77.57
62.13
55.94
59.45
63.35
generalization to novel objects from seen categories, we randomly split the data for each task M into 80% training and 20% testing. To test generalization unseen object categories, conduct separate experiment for each target category of the function objects, where we withhold data involving the target category and train on the remaining data. Details of the evaluation are discussed in Appendix D. We present results with binary success as metric in Appendix E.
5.2 Compared Methods
Baselines. We compare CPM with four main baselines. The first is Transformer-BC, which uses a multimodal transformer encoder-decoder from prior work [30] to condition on point clouds of the objects and autoregressively predict target transformations. The second baseline is based on TAX- Pose [37] which predicts relative poses between two objects from point-wise soft correspondences. The third is PC-DDPM; similar recent work [40, 47], conditional denoising diffusion probabilistic model [53] is trained to predict target transformations based on input point clouds of both the function and the anchor objects. The fourth baseline is the Part-Awart PC-DDPM, which takes in both point clouds of the objects and per- per-point segmentation masks that indicate object parts. We discuss the baseline implementations in details in Appendix B.
CPM variants. We evaluate several variants of model. The first is DDPM with 6D rotation representation instead of SE(3). This variant of CPM learns different diffusion models for different parts. However, it does not compose pose predictions in different part frames. This model is directly adapted from existing composable diffusion models for image generation [48, 49]. The second is DDPM with training-time composition; this model jointly train all primitive diffusion models by composes thier noise predictions at training time. The last group are the individual primitive diffusion models, which use single DDPM models corresponding different part-part correspondences. without any composition.
5.3 Simulation Results
Comparisons baselines. We evaluate CPM's generalization capability in two settings. First, Table shows comparison of generalization to novel objects from seen categories. Overall, our model achieves strong performance on both tasks of "pouring" and "safe placing". We note that TAX-Pose struggles with pouring that requires modeling multimodal actions because the method extracts a single relative pose estimate from a fixed set of correspondences. The autoregressive Transformer--H is also not enough to capture the full distribution of the pouring action. We note that although Parr- PC-DDPM leverages the same part segmentation as CPM, it fails to achieve stronger performance compared the PC-DDPM baseline, which only uses the object point clouds as input. We attribute this to its potential overfitting the part segmentations within the training data. By contrast, CPM is able to effectively leverage part segmentations by learning primitive diffusion models and composing them at inference time. Our model shows substantial improvements in the

---

Table 3: We ablate the contributions of CPM the ahility to generalize to novel categories of objects.
Target Pose Rep
Part Frames
Composition
Pouring
Safe Placing
6D + Trans
No
Inf-time
71.22
68.77
SE(3) SE(3)
Yes
Train-time
69.89
48.46
Yes
Inf-time
75.11
59.58
Table 4:
We explore the effect of composition, comparing individual diffusion models, in generalization across both "pouring" and "safe placing" tasks. "We note that for the align and facing-uy up evaluation, small percentage examples were removed as they not contain the involved parts in the partial object point clouds.
Pouring
Safe Placing
(align, rim, rim)
70.05*
(contain, head, body) (touch, tip, bottom)
41.22
(facing-up, handle, body) (tilt, body, body)
16.42*
9.34
68
(place, body, body)
39.86
CPM
75.11
CPM
59.58
"safe placing" task compared to other diffusion- basse methods, largely due to each part constraint significantly restricting the target pose distribution in this task. For instance, the constraint that requires the tip of the function object to touch the bottom of the anchor object effectively constrains the target pose.
Our second set of experiments assesses the model's capacity to generalize to unseen object categories, thereby highlighting the efficacy of part-based correspondences. Results can be found in Table 2. Remarkably, CPM demonstrates its capability to generalize across object categories for both tasks in a zero-shot manner. CPM's s performance dips slightly for pans as the rim's 's of pans are significantly larger compared to rim's s encountered during training (for example, those of bowls and mugs). As comparison, all baselines fall short in consistently generalizing new categories for both tasks. TAX-Pose is not able to maintain strong performance for safe placing when generalizing more geometrically complicated objects including scissors and forks. Our methods robust to changes in local geometry and overall topology by leveraging compositions of part-based correspondences.
Ablation. First, we assess the significance of our SE(3) encoding, part frame-based transfor- mation, and inference-time composition within the context of generalizing unseen categories of objects. As depicted in Table 3, our full CPM with part frames and inference-time composi- tion shows superior performance compared to the model trained with training-time time composi- tion. This verifies the importance of our designs to support part-based composition and gener- alization. Compared to the variant based 6D Rotation 3D Translation encoding, CPM yields better performance on the pouring task, a scenario where the rotation of the function ob- ject plays a pivotal role. On the safe placing task, which involves less rotation of objects, we observe a more comparable performance with our model. These results highlight the importance of SE(3) diffusion model in rotation prediction.
Body
Contain Frod
TI Touch Boltom
Composed
Figure 4: We illustrate the learned distribution of each primitive diffusion model, which generates diverse sam- ples conforming the specified constraints. well as the distribution from the combined full CPM model. The highest-ranked sample is highlighted.
Second, we compare the performance of composed part-based diffusion models with the performance of primitive diffusion models. Shown in Table 4, the composed model outperforms individual diffusion models, showing the efficacy of our composition paradigm. In addition, these results show the importance of different part-based constraints for the given tasks. the "pouring" task, align and tilt strongly constrain the target pose for the function object, while for the "safe placing" task, contain and place constraints more salient. Fig. 4 provides qualitative visualization by showcasing the part-conditioned distribution associated with each individual diffusion model for
7

---

Figure 5: We show sampled frames from trajectories of CPM's policy. The model is trained only on demonstra- tions with pans, bowls, and wine glasses in simulation and generalizes to mugs in the real world.
various constraints, as well as the corresponding composed distribution. The quantitative performance of contain and place primitive models for these tasks aligns with this qualitative comparison, as they have learned distributions that are close to the composed model. The CPM paradigm allows us to train each primitive diffusion model independently, encouraging each model concentrate on distinct functional affordances, thus enabling them to learn and generalize to diverse distributions of samples. During inference, the composition of distributions learned by individual models enables CPM to find solutions that satisfy all correspondence constraints.
5.4
Transfer
Finally, we show real-world robot manipulation experiment for the "pouring" task, highlighting the transferability our CPM real-world manipulation. In this setting, we use the primitive diffusion models trained simulation data with function objects of glasses, pans, and bowls, and zero-shot transfer mugs in the real-world experiment. Our setup includes Franka Emika robot mounted in a tabletop environment. To conduct pouring, we perform plane segmentation and k-means clustering to extract object point clouds from the scene point cloud captured by two calibrated Azure Kinect RGB-D cameras. Next, we apply pre-trained point transformer (PT) model [58] for part segmentation. The segmentation model is trained on simulation data only. We then apply CPM trained in simulation for the pouring task. To execute the trajectory. we use the Contact-Grasse
[59] sample robot grasps the function object and Operational Space Controller [60] with impedance from Deoxys [60] to following sequence of end-effector pose waypoints computed from the target transformations. Figure 5 shows our real-world setup and example trajectories predicted by CPM on unseen mugs with different shapes and sizes.
6 Limitations and Conclusion
We introduced composable part-based manipulation (CPM), as an approach that leverages object-pan partt decomposition and part-part correspondences for robotic manipulation. We show that representing actions combinations of constraints between object parts enables strong generalization. Through the composition of primitive diffusion models, we gain generalization capabilities across novel instances of objects as well as unseen object categories, in simulation and in real-world robot experiments.
In this paper, we focus on manipulation tasks involving two objects. Extending CPM to learn skills involving more objects would be important for future work, particular for manipulating piles or stacks of objects. Second, poses. Extending the transformer- longer trajectory is important for potentially wider range of tasks. In addition, CPM does not model temporal constraints over the trajectory. One possible extension is to learn trajectory samplers for temporal constraints and trajectories with loops. CPM assumes external part segmentations. Although many categories can be segmented by off-the-shelf computer vision models [26], extending the system to jointly learn finetune part segmentation is important. Finally, composing larger number of diffusion models may require more efficient sampling techniques such [61]. We provide an extended discussion of CPM's assumptions in Appendix F and suggest directions for future research.
parameterize each manipulation action by the starting and ending diffusion model to output more waypoints parameterize

---

Acknowledgments
We extend our gratitude to the members of the NVIDIA Seattle Robotics Lab, the RAIL research lab Georgia Tech, and the Stanford Vision and Learning Lab for insightful discussions. This work is in part supported by NSF grant 2214177, 221 1258, AFOSR grant 19550-22-1-02493 FA9550-23-1- 0127, ONR MURI grant 100014-22-1-2111 1-22400 the Stanford Institute for Human- Centered Artificial Intelligence (HAT), the MIT-IBM Watson Lab, the MIT Quest for Intelligence, the Center for Brain, Minds, and Machines (CBMM, funded by NSF STC award SCFF-12310 (23), and Analog Devices, JPMC, and Salesforce. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do necessarily reflect the views of our sponsors.
References
[1] C. Paxton, C. Xie, T. Hermans, and D. Fox. Predicting Stable Configurations for Semantic Placement of Novel Objects. In CoRL, 2021. 1,2
[2] W. Liu, Du, T. Hermans, S. Chernova, and C. Paxton. StructDiffusion: Language-Guided Creation of Physically-Vaiie Structures using Unseen Objects. In RSS, 2023, 1, 2,
[3] Y. Huang, A. Conkey, and T. Hermans. Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers. In ICRA, 2023. 1,2
[4] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Péree Integrated Task and Motion Planning. Annual Review of Control, Robotics, and Autonomous Systems, 4:265-293, 2021.
[5] K. Mo. L. J. Guibas, M. Mukadam, A. Gupta, and S. Tulsiani. Where2Act: From Pixels to Actions for Articulated 3D Objects. In CVPR, 2021. 2
[6] Z. Xu, Z. He, and S. Song. UMPNet: Universal Manipulation Policy Network for Articulated Objects. RA-I., 2022. 2
[7] R. Wu, Y. Zhao, K. Mo, Z. Guo, Y. Wang, T. Wu, Q. Fan, X. Chen, L. Guibas, and H. Dong. VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects. In ICLR, 2021.
[8] H. Geng, Xu, C. Zhao, C. Xu, L. Yi, Huang, and H. Wang. GAPartNet: Cross- Categoyy Comain-Generaliznn Object Perception and Manipulation via Generalizable and Actionable Parts. In CVPR, 2023. 2
[9] J. Alcotti and S. Caselli. Manipulation Planning of Similar Objects by Part Correspondence. In ICRA, 2011.
[10] N. Vahrenkamp. L. Westkamp, N. Yamanobe, E. E. Aksoy, and T. Asfour. Part-based grasp planning for familiar objects. In IEEE-RAS International Conference Humanoid Robots (Humanoids), 2016. 2
[11] Parashar, J. Vakil, S. Powers, and C. Paxton. Robot Learning. arXiv:2304. 11.00...113.... 2023. 2
Attention Policies for Efficient
[12] O. Mees, J. Borja-Diaz, and W. Burgard. Grounding Language with Visual Affordances Unstructured Data. In ICRA, 2023.
[13] E. Valassakis, G. Papagiannis, N. Di Palo, and E. Johns. Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learning. IROS, 2022.
[14] A. Myers, C. L. Teo, C. Fermüller, and Y. Aloimonos. Affordance Detection of Tool Parts from Geometric Features. In ICRA, 2015. 2, 4

---

[15] T.-T. Do, A. Nguyen, and I. Reid. AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection. In ICRA, 2018. 2, 4, 16
[16] S. Deng, X. Xu, C. Wu, K. Chen, and K. Jia. 3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding. In CVPR, 2021. 2,4, 16
[17] W. Liu, A. Daruna, and Chernova. CAGE: Context-Aware Grasping Engine. In ICRA, 2020.
2
[18] Ardón, É. Pairet, R. P. Petrick, S. Ramamoorthy, and S. Lohan. Learning Grasp Affordance Reasoning through Semantic Relations. In IROS, 2019.
[19] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kPAM: KeyPoint Affordances for Category- Level Robotic Manipulation. In ISRR, 2022. 2
[20] Z. Qin, K. Fang, Y. Zhu, L. Fei-Fei, and S. Savarese. KETO: Learning Keypoint Representations for Tool Manipulation. In ICRA, 2020. 2
[21] D. Turpin, L. Wang, Tsogkas, S. Dickinson, and Garg. GIFT: Generalizable Interaction-
aware Functional Tool Affordances without Labels. In RSS, 2021. 2
[22] L. Manuelli, Y. Li, P. Florence, and R. Tedrake. Keypoints into the Future: Self-Supervised Correspondence Model-Based Reinforcement Learning. CoRI,, 2020. 2
[23] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural Descriptor Fields: SE(3)-Fquivariant Object Representations for Manipulation.
In ICRA, 2022. 2
[24] E. Chun, Y. Du, A. Simeonov, T. Lozano-Perez, and L.. Kaelbling. Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulation. In ICRA, 2023.
[25] J.-S. Ha, D. Driess, and M. Toussaint. Deep Visual Constraints: Neural Implicit Models for Manipulation Planning from Visual Input. RA-L, 108:00887410864, 2022. 2
[26] M. Liu, Y. Zhu, H. Cai, S. Han, Z. Ling, F. Porikli, and H. Su. PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models. In CVPR, 2023.
8, 16
[27] D. Hadjivelichkov, S. Zwane, L. Agapito, M. Deisenroth, and Kanoulas. One-Shot Transfer of Affordance Regions? AffCorrs! CoRL, 2022. 2,
(28) W. Goodwin, Havoutis, and I. Posner. You Only Look at One: Category-Level Object Representations for Pose Estimation From Single Example. In CoRL, 2022. 2
[29] W. Yuan, C. Paxton, K. Desingh, and D. Fox. SORNet: Spatial Object-Centric Representations
for Sequential Manipulation. In CoRL, 2021. 2
[30] W. Liu, C. Paxton, T. Hermans, and D. Fox. StructFormer: Learning Spatial Structure for
Lunguage-Guided Semantic Rearrangement of Novel Objects. In ICRA, 2022. 2, 6, 14
[31] M. Shridhar, L. Manuelli, and D. Fox. CLIPort: What and Where Pathways for Robotic Manipulation. In CoRL, 2021.
[32] T. Silver, R. Chitnis, Tenenbaum, L. P. Kaelbling, and T. Lozano-Pérez. Learning Symbolic Operators for Task and Motion Planning. IROS, 2021.
[33] K. Kase, C. Paxton, H. Mazhar, T. Ogata, and D. Fox. Transferable Task Execution from Pixels through Deep Planning Domain Learning. In ICRA, 2020. 2
[34] K. Mo, Y. Qin, F. Xiang, H. Su, and L. Guibas. O20-Afford: Annotation-Fre Free Large-Scale Object-Object Affordance Learning. In CoRL, 2022. 2
10

---

[35] J. Liang and A. Boularias. Learning Category-Leve Manipulation Tasks from Point Clouds with Dynamic Graph CNNs. In ICRA, 2023.
[36] D. Driess, J.-S. Ha, M. Toussaint, and Tedrake. Learning Models as Functionals of Signed- Distance Fields for Manipulation Planning. CoRL, 2022. 3
[37] C. Pan, B. Okorn, H. Zhang, B. Eisner, and D. Held. TAX-Pose: Task-Specific Cross-Pose
Estimation for Robot Manipulation. In CoRL, 2022. 3, 6
[38] D. Seita, Y. Wang, S. J. Shetty, E. Y. Li, Erickson, and D. Held. ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds. In CoRL, 2022. 3
[39] M. Janner, Y. Du, Tenenbaum, and Levine. Planning with Diffusion for Flexible Behavior Synthesis. In ICML, 2022. 3
[40] J. Urain, N. Funk, J. Peters, and G. Chalvatzaki. SE(3)- Difuusiifusion::illd: Learning Smooth Cost Functions for Joint Grasp and Motion Optimization through Diffusion. In ICRA, 2023. 3, 4, 6, 14
[41] S. Huang, Z. Wang, Li, Jia, T. Liu, Y. Zhu, W. Liang, and S.-C. Zhu. Diffusion-Based Generation, Optimization, and Planning in 3D Scenes. In CVPR, 2023. 3
[42] I. Kapelyukh, V. Vosylius, and E. Johns. DALL-E-Bot. Introducing Web-Scale Diffusion Models to Robotics. RA-L, 2023. 3
[43] A. Ajay, Y. Du, A. Gupta, Tenenbaum, T. Jaakkola, and Agrawal. Is Conditional Generative Modeling all you need for Decision-Making! In ICLR, 2023.
[44] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang. J. Singh, C. Tan, J. Peralta, B. Ichter, et al. Scaling Robot Learning with Semantically Imagined Experience. arXi::232. 1550, 2023. 3
[45] U. A. Mishra and Y. Chen. ReorientDiff: Diffusion Model based Reorientation for Object Manipulation. arXiv:2303. 000223333/700.. 12700, 2023. 3
[46] C. Higuera, B. Boots, and M. Mukadam. Learning Read Braille: Bridging the Tactile Reality Gap with Diffusion Models. arXiv:2304.01182, 2023. 3
[47] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS, 2023. 3, 6, 14
[48] N. Liu, Li, Du, Torralba, and J. B. Tenenbaum. Compositional Visual Generation with Composable Diffusion Models. In ECCV, 2022, 3, 4,
[49] Y. Du, C. Durkan, R. Strudel, J. B. Tenenbaum, S. Dieleman, R. Fergus, J. Sohl-Dickstein, A. Doucet, and W. Grathwohl. Reduce, Reuse, Recycle: Compositional Generation with Energy-Bin Based Diffusion Models and MCMC. In ICML, 2023. 3,4,
[50] Z. Huang, K. C. Chan, Y. Jiang, and Z. Liu. Collaborative Diffusion for Generation and Editing. In CVPR, 2023. 3
Face
[51] R. Xu, F-J. Chu, C. Tang, W. Liu, and P. A. Vela, An Affordance Keypoint Detection Network for Robot Manipulation. RA-L, 602)22870- 2021. 4
[52] M.-H. Guo, J.-X. Cai, Z.N. Liu, T.-J Mu, R. R. Martin, and S.-M. Hu. PCT: Point Cloud Transformer. Computational Visual Media, 2021. 4, 13, 14
[53] J. Ho, A. Jain, and P. Abbeel. Denoising Diffusion Probabilistic Models. In NeruIPS, 2020. 14

---

[54] K. Mo, S. Zhu, A. X. Chang, L. Yi, S. Tripathi, L. J. Guibas, and H. Su. PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding. In CVPR, 2019. 5, 14
[55] M. Savva, A. X. Chang, and P. Hanrahan. Semantically-Ennichhe 3D Models for Common-sense Knowledge. In CVPRW, 2015. 5, 14
[56] C. Eppner, A. Mousavian, and Fox. ACRONYM: Large-Scale Grasp Dataset Based Simulation. In ICRA, 2021. 5, 14
[57] E. Coumans and Y. Bai. Pybullet, A Python Module for Physics Simulation in Robotics, Games and Machine Learning, 2017. 5
[58] H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun. Point Transformer. In ICCV, 2021. 8
[59] M. Sundermeyer, Mousavian, Triebel, and D. Fox. Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes. In ICRA, 2021.
[60] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. VIOLA: Imitation Learning for Vision-Based Manipula- tion with Object Proposal Priors. In CoRL, 2022.
[61] Q. Zhang and Y. Chen. Fast Sampling of Diffusion Models with Exponential Integrator. In ICLR, 2022. 8
12

---

A Network Architecture
For each functional correspondence PA.j. PA..PPE.ii PF,)) we aim to learn a generative distribution Pe.s, (Tp,al : Here we discuss the network architecture for primitive diffusion model <0.8; that learns to estimate the generative distribution. We leverage modality-speciffe encoders convert the multimodal inputs to latent tokens that are later processed by transformer network.
Object encoder. Given part point clouds XPAS and Pp we use a learned encoder hp to encode each part separately as (pp,)A.)) and hp (pp,.)) This encoder is built on the Point Cloud Transformer (PCT) [52].
Diffusion encodings.
Since the goal transformations Tp3 (T are iteratively refined by the diffusion model and need to feed back to the model during inference, we use MLP to encode each goal transformation separately (TT,P),.)) To compute the time-dependent Gaussian posterior for reverse diffusion, we obtain a latent code for t using a Sinusoidal embedding ht)me
Positional encoding. We learned position embedding (4) indicate the position index of the part point clouds and poses in input sequences to the subsequent transformer.
Diffusion Transformer. The diffusion model predicts the goal poses Tp, starting from the last time
(0)
step of the reverse diffusion process T (O, which is sampled from multivariate normal distribution with independent components. We use a transformer encoder as the backbone for the diffusion model 60,s; ((T()) PT Wi
XPAs x which predicts the dipendenn noise (2) We obtain the transformer input for the parts x and the target poses T as {e)" ..... N
A [hp(X PA, Pos (0); htime (t))
(4)
(e) [Np( Pr.u): lpos (1); htime (4)
X
(4) (T(/ Pje..)) poo (n 2); hasme
where [:] is the concatenation at the feature dimension. The model takes in the sequence (2) (t) (1) and predicts (c(²) ,..., (t) ( for the object poses.
(xa) :XF TN...TT
Parameters. We provide network and training parameters in Table A1.
Table A1: Model Parameters
Parameter
Value
Number of PA.j and PF. points
512
PCT point cloud encoder hp out dim
200
Position embedding 1pos
learned embedding
Position embedding lpos dim
16
Time embedding htmme
Sinusoidal
Time embedding htmme dim
40
Pose encoder ht out dim Transformer number of layers Transformer number heads Transformer hidden dim Transformer dropout Diffusion steps T
200
4
128
0.0
200
Diffusion noise schedule Br Start value So
Linear
0.0001
End value 3r
0.02
Loss
Huber
Epochs
2000
Optimizer
Adam
Learning rate
le-4
Gradient clip value
1.0
13

---

B Implementation Details for Baselines
We discuss the implementation of each baseline below:
Transformer-Be this baseline takes point clouds X and XF input, and predicts target transformations TAP. This baseline uses multimodal transformer encoder-decoder from prior work [30] to condition on point clouds of the objects and autoregressively predict target transformations. The point clouds are first individually encoded with a point cloud transformer [52]. The point cloud embeddings are fed to the transformer encoder. The transformer decoder autoregressively decodes the target poses (T,F, TAF.N
TAX-Pose: this baseline takes point clouds XA and XF as input, and predicts target transformations TAF. We use the code and hyperparameters from the official repository We use the variant that does not require pretrained object embeddings because we use different objects from the paper. As discussed in Appendix F.1.2 of the original paper, pretraining mainly helps to reduce training time. Because the TAX-Pose model only predicts one relative pose for each pair of point clouds, we learn a separate model for each transformation in TAF. Specifically, one TAX-Pose is trained predict start pose and another Poxe is trained to predict end pose.
PC-DDPM: this baseline takes point clouds XA and XF as input, and predicts target transformations TAP. Similar to recent work [40, 47], conditional denoising diffusion probabilistic model [53] is trained predict target transformations based input point clouds of both the funetion and the anchor objects. This model has the same architecture, including encoders, latent embeddings, and the diffusion transformer, as the primitive diffusion models, which is discussed in Appendix A.
Part-Aware PC-DDPM: this baseline takes point clouds XA E R- and XF R- x3 and two segmentation masks IA R and Ir target transformations TAF- Nx is the number of points for each point cloud and N₂ is the number of known object parts. Each channel of the segmentation mask is binary mask indicating points for a specific part. Each segmentation mask encodes all parts that can be extracted from an object point cloud. For simulation experiment, the segmentation masks come from groundtruth part segmentation. While CPM use the segmentation masks extract part point clouds, this baseline directly encode the segmentation mask together with the object point cloud, This baseline shares most of the network architecture as PC-DDPM except that point cloud encoder now encodes [A] E I (x++Nx) and [x;; IF] E Rx (3+Nr))
as input, and predicts
C Dataset Details
In total, we collected 4522 successful demonstrations for pouring and 2836 successful demonstrations for safe placing. For each experiment, and the remaining data for initializing the simulation. We provide a breakdown of the dataset in Table A2. Because the expert policies do not have 100% success rate, the models will only be trained on the successful demonstrations. Below we discuss our data collection process in details.
subset of these demonstrations for training the models,
Sourcing 3D objects. We source a wide variety of 3D objects from PartNet [54] and the subset of ShapeNetSem [55] objects categorized in the Acronym dataset [56]. We use 13 object categories to investigate generalization, including mug, pan, bowl, wine glass, knife, can opener, scissors, screwdriver, fork, spoon, marker, pen, and flashlight. Some object categories are reused for different tasks; for example, mug is used as an anchor for safe placing but also as an object for pouring.
Extracting aligned parts. Our generative diffusion models uses part segmentations of objects to learn primitive diffusion models. For 3D objects from PartNet, we use the segmentations provided in
"Code from https://gtthu..com// pawwaa/.e......aaapa...ee....
14

---

Table A2: Simulation and Demonstration Data
Task
Object
Source
Number of Simulations Number Success Demonstrations
Pen
PartNet
1000
568
Fork
PartNet
1000
390
Screw Driver
PartNet
1000
145
Spoon
PartNet
1000
410
Safe Placing
Knife
Acronym
1000
496
Scissors
PartNet
1000
354
Flashlight
PartNet
1000
141
CanOpener
PartNet
1000
101
Marker
PartNet
1000
231
Mug
PartNet
2000
1051
WineGlass
PartNet
2000
1542
Pouring
Bowl
Acronym
2000
776
Pan
PartNet
2000
1153
the dataset. For 3D objects from ShapeNetSem, we first label 3D keypoints, then from the labeled keypoints, procedurally extract parts. As ShapeNet provides canonical poses for 3D models, can also align the extracted functional parts for each object category.
Simulating trajectories and rendering. We simulate the robot-object interactions by tracing the trajectories defined by the parameters. We first use multiple cameras to render RGB-D images, which yield realistic object point clouds. We then map the functional parts to the point clouds with the correct transformation and scaling. Finally, Because these parts extracted from the rendered point clouds, they can be incomplete, which increases the robustness of our method and helps transferability to real-world settings.
obtain point cloud segments of each affordance part.
D Evaluation Details
In Section 5, we report task completion scores. For each experiment, we randomly draw 100 samples from the withheld testing data to initialize simulation for evaluation. This procedure ensures that the action can be successfully performed for the pair of anchor and function objects. To systematically evaluate multimodal actions (e.g, pouring from different directions), sample from each model times and simulate the predicted actions. We repeat each experiment with 3 different random seeds, resulting in total of 1500 trials.
The task score indicates task completion between failure (0) and success (100), with credits assigned for partial completion. The score is computed based on model-based classifiers designed for each task. Now describe how the score is computed in more detail:
Pouring: first use PyBullet's collision test to check whether the function object and anchor object will ever interpenetrate during the execution of the action by rigidly transforming the function object to the predicted poses. If the objects interpenetrate, we assign a score of zero because the action cannot be realistically executed. Then we simulate the pouring action, and use the percentage of particles successfully transferred from the function object to anchor object as the partial score.
Safe Placing: similar to pouring, we check interpenetration for the start pose of the placement action. If the objects interpenetrate, we assign a score of zero. Then we simulate the placement action until contact between the anchor and function object. the orientation of the function object is incorrect (e.g.. the blade of the knife is outside of the container), we assign score of zero. If the orientation correct, the percentage of the trajectory parameterized by the predicted transformations that is successfully executed is used as the partial score.
15

---

E Additional Results
Besides reporting the task completion scores, we include additional task success rates Table A3 and Table A4. For pouring, a trial is considered successful if there is no interpenetration between objects and 70% of particles are successfully transferred. For safe placing, successful trial requires no interpenetration the predicted start pose for the function object, correct orientation of the function object, and 70% of the predicting trajectory being successfully executed without collision between objects. We observe similar trends as the results presented in Section 5.
Table A3: CPM shows strong generalization to novel instances of objects within seen categories.
Model
Pouring
Safe Placing
Transformer-BC
17.53=3.13
33.2742.14
TAX-Pose
21.33=0.58
7..00++1.00
PC-DDPM
70.6711.27
48.73:12.97
Part-Aware PC-DDPM
73.60-12.60
36.53:12.220 68.871:2.25
CPM (ours)
1..87
Table A4: CPM demonstrates strong generalization to function objects from unseen object categories.
Pouring
Safe Placing
Model
Bowl
Glass
Mug
Pan
Fork
Pen
Scissors
Transformer BC
10.00:42.55
19.20=0.92
5.80-:11
29.33-42.004
24.00=1.51
27.33422334
18.47:42.93
TAX-Pose
21.00
3.00+:1.00
8.00-11000
42.67442.08
47.67=3221
62.67:4.04
33.33::1.15
PC-DDPM
56.53-+2.000 00
70.67+33.06
68.67++-.31
59.93-+2.80
38.00=3.83
43.47+1.68
28.47+00.3
Patt-Aare PC-DDPM
54.87:12.10
68.33-42.97
65.20:14.61
62.00:13..6
28.67=1..8
42.40+3.12
17.6742.70
CPM (ours)
76.40:11.78
78.93=3.14
56.00:15.26
54.67:11.0
53.93-2991
56.5342.04
62.07:11722
Assumptions
During training, our method assumes 1) description of the manipulation skill of part correspondences, 2) access the dataset of successful trajectories, and 3) access to part segmentations for objects in the dataset. During testing, being manipulated. We contend that these assumptions align with our current focus. Nonetheless, subsequent research should aim to address them.
method assumes the part segmentations for objects
First, the description of manipulation skills symbolic text, e.g., pouring from mugs bowls contains three constraints. They can be easily annotated by humans as there no need to specify any continuous parameters mathematical formulas. An interesting future direction is leverage large language models to more efficiently extract constraints. CPM then learns the grounding of these constraints from data.
Second, we assume access to successful manipulation trajectories. That is, we do not assume any additional annotations, such as programs for generating these trajectories. The key focus of the paper improve the data efficiency of learning such skills, particular for generalization across categories. An important future direction is to improve the data efficiency of this method and learn from noisy human demonstrations.
Finally, relying on external part segmentation is limiting, but 2D or 3D part segmentation models are generally available for many object categories [15, 16, 26]. An exciting future direction is to extend the current framework automatically discover functional part segmentations leveraging manipulation data.
16


Reviews:
[{"review_id": "m7H6Fa8efPA", "reviewer": "Reviewer_6mc4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "\nThe paper proposes an approach to behavior cloning for manipulation. The manipulation behavior is generated by composing a small set of primitives. Each primitive is defined with respect to object parts, i.e. the rim of a bowl or the handle of a mug. The primitives are learned using behavior cloning. The system is presented with a database of expert trajectories that are labeled with behavior and with the parts involved. A diffusion model is trained to reproduce the behaviors, parameterized by the shape of the objects (encoded as a point cloud) and the task. A key aspect of the work is that multiple primtives can be combined at test time. This is done simply by adding the de-noising noise from all to-be-combined primitives. The resulting trajectories are expected to achieve the objectives of each primitive.\n", "summary_of_recommendation": "I think this is a strong paper, but probably not Earth shattering. I think we should accept, but if another reviewer is more knowledgeable and feels it should be rejected, I'm fine with that.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "None", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nI think the general approach here is interesting. There are a couple of key ideas: 1) creating parts-based primitives rather than object-based primitives; 2) combining diffusion policies by simply adding. I think the parts-based approach is a little unusual in the literature and makes sense and I'm happy to see it here. Similarly, the approach to combining policies is also interesting. However, I'm not at all clear on how effective the adding approach would be -- the concern would be that multiple objectives cancel each other out. It would be good to explore this question in future work.\n\nThere is a comparison to PC-DDPM and part-aware PC-DDPM. I don't know if there are other methods that should be comared against, but it seems like at least some kind of baseline is being done here.\n\nI'm also disappointed that the parts need to be hand labeled instead of being discovered. It does seem like there is a lot of hand coded information here. Nevertheless, the overall ideas are novel and I'm not super concerned about those points.\n\nOverall, I think this is a decent paper that has interesting ideas.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "cHwkEMDQVaE", "reviewer": "Reviewer_djYt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an approach for learning robot manipulation skills and generalising them to unseen objects from simulated demonstration data. The approach learns several diffusion models for a specific task, where each diffusion model is conditioned on only a single part of the function and anchor object and is responsible for capturing the distribution of valid function object poses for that task. For example, one diffusion model could be conditioned on the point cloud of the rim of a mug and bowl, while another could be on the bowl's body and the mug's handle. At test time, the output of all diffusion models is combined to predict the starting and ending pose for the function object, which, when realised, would complete the task.", "summary_of_recommendation": "The proposed method requires much manual engineering to set up a simulated pipeline for obtaining demonstrations. Moreover, the paper is very difficult to follow, and the experiments section is unclear, making it difficult to assess how well the method works and how it was evaluated. Moreover, it is difficult to understand how the method was deployed in the real world, given that each diffusion model makes predictions in the coordinate frame of the considered object part, which might not be annotated for novel real-world objects.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "**Training Data**: \n- For each object category/instance, do you require a separate frame of reference for each of the considered parts of an object, e.g. for the rim, body and handle of a mug? If so, how do you obtain these frames of reference in the real world when encountering a novel object class?\n- What is the exact form of the training data in the context of a single demonstration? Is this a (full) point cloud of each considered part of the anchor and function object, alongside with the final pose of the function object with respect to the anchor object? If the point cloud is full and not partial, how is this solved in the real world?\n\n**Training and the model**:\n- How many demonstrations are required to train the model?\n- What is the architecture of the model?\n\n**Experiments**:\n- How exactly did you evaluate the methods in simulation? That is, how many evaluations did you do, how many object categories did you consider etc?\n- Do you have any qualitative results for the real world experiment?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Although the method is interesting, it appears to require a lot of work, engineering and prior knowledge to get it working for a novel skill. For example, consider the skill of pouring. \n\nFirstly, all demonstrations of the pouring skill must be coded in simulation. This requires engineering a heuristic for the pouring skill that will be compatible with many different function and anchor object categories and will work for multiple pouring tasks, such as: pouring from a mug into a bowl, from a pan into a bowl, from a wine glass into a container etc. Having coded such a heuristic that works reliably enough to generate multiple demonstrations involving different object categories is already equivalent to coding a policy capable of reliably executing the skill, somewhat reducing the need for the proposed method.\n\nSecondly, all demonstrations must be annotated with part-based functional correspondences. For example, for the pouring skill this would be <align, rim(mug), rim(bowl)>, <tilt, body(mug), body(bowl)> and <facing_up, handle(mug), body(bowl)>. Decomposing a general skill into these part-based functional correspondences is additional manual work that must be carried out for each new skill we would like to teach a robot. \n\nLastly, based on the part-based functional correspondences, the relevant object parts for the skill must be extracted in all demonstrated trajectories. Furthermore, during inference, they must also be segmented reliably.  \n\n**Strengths**:\n- Simultaneously predicting a motion from different parts of a function and anchor object and combining the predictions is quite appealing.\n\n**Weaknesses**\n- The proposed approach requires a lot of engineering for each new skill. Namely, the biggest issue is providing demonstrations in simulation for a given skill involving different object categories for the anchor and function object. On the one hand, the data generation process could be automated. However, after doing this, we no longer have a need to learn the skill as we already have a policy capable of executing it. On the other hand, we could collect the demonstrations manually, which would be very time-consuming.\n- The proposed approach requires a lot of prior knowledge, such as part-based functional correspondences for each skill and a separate frame of reference for each considered part of the object.\n- Representing general skills as a starting and ending pose is quite limited. For example, for the pouring task, the robot may spill some of the liquid as it approaches the anchor object without any constraints on the orientation of the function object during the approach phase.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "_5OHOJy1Pv9", "reviewer": "Reviewer_3XuZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes composable part-based simulation – it’s tackling the problem of learning and genralizing robot manipulation skills through part-part correspondences. The method proposes the use of composable diffusion models which capture the various inter-object relationships. \n", "summary_of_recommendation": "Overall, the method is interesting and appears useful, but there are some unanswered questions. I am recommending a weak acceptance as I believe the concerns are all addressable, but I can ultimately be convinced either way depending on the author’s responses and the comments of other reviewers.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "How are the constraints defined in the context of this work?\n\nL166 ``’the composition corresponding to maximize’ - what does this mean?\n\nWhat tasks were considered for testing this method, were there any that were not considered due to limitations of this particular model?\n\nL199 claims that rendered point clouds can be noisy – was artificial noise applied to the point clouds? Typically, simulated point clouds are as ``perfect’’ as possible. \n\nL185 – how are the parts procedurally extracted?\n\nWhile it is not solving the problem in the same way, is it not appropriate to compare the method with approaches such as TAX-Pose and R-/NDF? Why/Why not?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## Strengths\n\nThe method leverages the inductive bias that parts and their language descriptions capture their functional affordances, formalizing an action as a set of rules relating to object parts that can be learned. This approach could be generalized to novel unseen instances without the need for massive datasets. \n\nThe related work is well structured and sufficient.\n\nPart-based methods allow easy generalization to objects with slightly different part structures (such as cups to mugs).\n\nAn interesting novel application of diffusion models to robotics.\n\nThe experimental results are convincing of the method's usefulness.\n\n## Weaknesses\n\nThe method assumes access to a set of symbolic constraints associated with each task. \nAdditionally, it is unclear how constraints, which are central for this method, are defined - are they specified by a user with specific parameters optimized by the model, are they fully pre-defined, are they fully learned?\n\nOnly two tasks are considered for generalization to unseen categories, of which the second is not particularly impressive. Other tasks that could have been considered include handled object on rack, gripper on various objects.\n\nIt is unclear why the method was not compared with other baselines such as TAX pose and NDF which tackle a similar problem\n\nThe limitations are only briefly discussed.\n\n## Other minor comments\n\nI suggest rephrasing the research contribution of the work. A research contribution is a novel and significant addition to the field of study that advances the existing knowledge, theories or practices. \n\nModelling manipulation based on part-part correspondences is not novel – many works within robot affordance learning research have done this (e.g. refer to the works of Eren Aksoy, or some of the references cited). Additionally, the third contribution is just a result of the other two.\n\nSome citations are capitalized wrong. \n\nL73: should this be “within” instead of “between”\n\nFigure 2 and its caption are unclear – How does the combined noise feed back into the sampling? \n\nWhere is the point cloud transformer in the figure (referred to on L138)? It has no clear start or end, and the images used to represent pose are confusing. \n\nL145 – T0 is different from the T0 used in the equation.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "KL7LOtlRuX", "reviewer": "Reviewer_ayCc", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an approach for learning manipulation skills which involve the interaction of two objects, with one object actively controlled by the robot. The key idea is to decompose objects into their constituent parts, and define skills in terms of the correspondences between parts. The authors propose learning separate diffusion models per correspondence, which generates initial and final poses for the actively controlled object conditioned on point-clouds. Skills involving multiple correspondences are then defined by the composition of the appropriate diffusion models. The authors demonstrate their approach on two tasks and show that the method can achieve generalization to unseen object categories. The authors also include a real-world demonstration of the method. ", "summary_of_recommendation": "I think this paper shows promise, but due to it missing some key details I do not believe it is ready for acceptance in its current form. If the manuscript was revised to improve clarity and add the key missing details (particularly the performance metric), I would be happy to change my recommendation to a weak accept. If the authors further incorporate an additional non diffusion baseline, I would change to a strong accept. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Revise the manuscript to improve clarity (from above section)\n- Incorporate additional non-diffusion baseline", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- Authors show real-world demonstration showing impressive successful sim-to-real transfer, including generalization to unseen mugs\n- The authors proposed method of using part-part correspondences as the basis of modelling manipulation actions, and the integration of this with diffusion models is very interesting and novel. I believe this paper is very relevant to the CoRL community. \n\nWeaknesses\nThe main weakness of the paper is that in places it is unclear and missing key details. In addition I believe there is room for improvement in the baselines used for comparison. \n\n- On line 135, and in the equation below line 144, the authors state that each individual takes as input two object point-clouds, but in the equation below line 160, these are now partial point-clouds. From the rest of the text, particularly the discussions of the baselines, I am inclined to think that the authors use partial point-clouds of the parts as input but it is unclear. \n- In the section 'Inference-time combination' the authors discuss performing the diffusion in SE3. This is further ablated in the experiments with the model with a 6D rotation parameterization. However, performing diffusion on SE3 is not a contribution of this paper, since that has been proposed in [40]. The section 'Inference-time combination' should be rewritten to make clear that this is from prior work and not part of the authors contribution. \n- I found the description of the baselines and their relationship to CPM very unclear. In particular, for both PC-DDPM and part-aware-PC-DDPM, it is very unclear to me what their input/output is. Is there a model per correspondence as with CPM, or a model per task? If neither, how are the correspondences/tasks encoded? This becomes especially important for part-aware-PC-DDPM because from the text I do not understand what the difference between part-aware-PC-DDPM and CPM is. \n- I cannot find anywhere in the text a definition of the performance metric. Is it a success rate? Over how many evaluations? The authors proposed approach shows improved performance including on unseen tasks according to this metric, but I cannot list this as a strength of the paper until I know what this performance metric means.\n- The authors baselines are only other diffusion models, and all of the baselines are developed by the authors. I think they would be much more convincing if they were contrasted from another method from the literature. Perhaps something from the offline RL literature such as CQL. \n- One key limitation that is not discussed in the limitations section is the quite restrictive set of tasks that this method can be applied to, namely consisting of two-object interactions with only one object actively manipulated. ", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel method based on object part decomposition and their correspondences to improve learning and generalization of robotic manipulation skills. All the reviewers recognize that the paper contains interesting ideas despite the fact that the current state of the approach requires a lot of manual engineering work. In the period of the rebuttal, the authors have provided careful answers with additional experimental results, e.g., two novel baselines,  to address the major questions of the reviewers. 3 reviewers suggest weak accept and one switches from strong reject to  strong accept after the rebuttal.
Decision: Accept (Poster)