Non-rigid Relative Placement through 3D Dense Diffusion
Authors: Eric Cai
Octavian Donca
Ben Eisner
David Held
Venue: CoRL 2024 (2024)
Keywords: Deformable
Non-rigid
Manipulation
Relative Placement
TL;DR: We expand the "relative placement" formulation beyond rigid body manipulation, allowing for generalization to novel object instances, unseen scene configurations, and multimodal placements for highly deformable tasks.

Abstract: The task of "relative placement" is to predict the placement of one object in relation to another, e.g. placing a mug on a mug rack. Recent methods for relative placement have made tremendous progress towards data-efficient learning for robot manipulation; using explicit object-centric geometric reasoning, these approaches enable generalization to unseen task variations from a small number of demonstrations. State-of-the-art works in this area, however, have yet to represent deformable transformations, despite the ubiquity of non-rigid bodies in real world settings. As a first step towards bridging this gap, we propose "cross-displacement" - an extension of the principles of relative placement to geometric relationships between deformable objects - and present a novel vision-based method to learn cross-displacement for a non-rigid task through dense diffusion. To this end, we demonstrate our method's ability to generalize to unseen object instances, out-of-distribution scene configurations, and multimodal goals on a highly deformable cloth-hanging task beyond the scope of prior works.

Non-rigid Relative Placement through 3D Dense Diffusion
Eric Cai, Octavian Donca, Ben Eisner, David Held Robotics Institute, School of Computer Science Carnegie Mellon University, United States {eycai, odonca, baeisner, dheld cheld)@andrer Candrew. cmu. edu
Abstract: The task of "relative placement" to predict the placement of one object in relation to another, e.g. placing a mug onto a mug rack. Through ex- plicit object-centric geometric reasoning, recent methods for relative placement have made tremendous progress towards data-efficient learning for robot manip- ulation while generalizing unseen task variations. However, they have yet represent deformable transformations, despite the ubiquity of non-rigid bodies in real world settings. As first step towards bridging this gap, we propose "cross- displacement" an extension of the principles of relative placement to geomet- ric relationships between deformable objects and present novel vision-bard based method learn cross-displacement through dense diffusion. To this end, we demonstrate our method's ability to generalize unseen object instances, out- of-distribution scene configurations, and multimodal goals multiple highly de- formable tasks (both in simulation and in the real world) beyond the scope of prior works. Supplementary information and videos can be found at our website.
Keywords: Deformable, Non-rigid, Manipulation, Relative Placement
Initial Observation
Goal Prediction through Displaispleemmett Diffusion
Successful Placement
Figure 1: Our method (TAX3D) uses dense displacement diffusion to determine how to perform a deformable eloth-hanginn task for unseen scene configuration.
Introduction
Learning from demonstrations has emerged as a powerful technique to imbue robots with complex manipulation capabilities. Recent approaches have achieved remarkable success on a wide variety of tasks, including mug-hanging [1, 2, 3], shelvine [4], cloth cloth-folding [5], and sauce-pouring [6]. One common approach for imitation learning is learn end-to-end visuomotor policies that map directly from observations to low-level robot actions; however, such approaches often struggle to generalize to novel variations of the scene (e.g, unseen object instances or object configurations). In this work, we aim to train robots to be robust to these variations for multimodal, deformable object placement tasks. Given demonstrations of a cloth-hanging task, for example, a robot should be able to successfully hang the cloth for unseen cloth instances or hanger positions. Similarly, the task has multiple possible goal configurations (e.g. multiple holes on the cloth through which to orient the hanger), the robot should be able output multimodal predictions complete the task.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

A simple solution is to collect an increasingly large dataset with all of the variations that one wishes to be robust to. However, collecting such dataset, particularly in the real world, may be pro- hibitively infeasible. Other recent works have taken an alternative approach by reasoning more explicitly about object geometry and casting the manipulation task as a "relative placement" prob- lem [1. 2, 4]. These approaches directly model the pose relationship between the object being manipulated (the "action object") and the rest of the scene (the "anchor") in an object-centric fash- ion by separately featurizing action and anchor geometries; these approaches have shown the ability to learn precise manipulation tasks with impressive generalization capabilities from small number (e.g. 10) of demonstrations [I]. However, these approaches also make strong assumptions about object rigidity, predicting an SE(3) transformation that does not apply to deformable objects.
In this paper, we present TAX3D (TAsk-Specific Cross- Pross-Displacement through Dense Diffusion), deep 3D-vision framework to handle multimodal goal prediction for deformable relative placement tasks (e.g. hanging cloth hanger). Namely, our contributions include:
(1) A formal definition of "cross- ccross-dispaccement in which we extend the idea of relative place- ment beyond rigid poses to arbitrary deformable settings through a dense representation.
(2) A novel method to predict cross-displanee coossdiispaaeement through object-centric point-wist diffusion.
A novel task benchmark for multimodal relative placement for deformable object manipu- lation from demonstrations, building DEDO [7].
We design experiments to rigorously evaluate performance under goal multimodality, out-of- distribution scene configurations, and variations in object geometry. Contrary to prior relative place- ment methods, we demonstrate that our method robust all of these variations for simulated and real-world cloth-hanging tasks, paving the way for generalizable non-rigid relative placement.
2 Related Work
Relative Placement for Object Manipulation: As previously described, there exists corpus of work that deconstructs placement task as relative pose prediction problem [8, 10, 11, 12]. Dense Object Nets (DON) [3] and Neural Descriptor Fields (NDF) [2] accomplish this by learn- ing dense embeddings and matching observation embeddings to demonstration embeddings. While these approaches work in constrained placement settings, they assume that the target object is moved relative to a static reference object. TAX-Pose [1] addresses this issue by directly modeling the rel- ative pose of objects in the scene and regressing task specispe "cross-pose" from learned dense embeddings. This inference pipeline, however, renders TAX-Pose unsuitable for multimodal goal prediction. Recent work (TAX-PoseD [13]) extends TAX-Pose to multimodal tasks using a ditional Variational AutoFncoder (cVAF) with dense spatially-grounded latent space. Relational Pose Diffusion (RPDiff) [4] leverages iterative pose de-noising to learn object-scent relationships that are multimodal, but diffuses directly in the space of SE(3) transformations. This fundamentally restricts RPDiff to rigid placement tasks. In contrast to these approaches, our method is able to handle both multi-modal placements and deformable objects for relative placement tasks.
Diffusion Models for Point Cloud Generation: Diffusion models [14, 15, 16, 17, 18, 19, 20] have emerged as the state-of-the-art in 2D image generation; their variational loss allows for more stable training and mode capture (in contrast to GANs), and the lack of aggressive regularization on their latent distribution mitigates the coverage-fidelifg tradeoff inherent to VAEs. Following this success, there have been many efforts to transfer these advantages 3D point cloud generation, either by applying 2D diffusion techniques to volumetric representations [21, 22] by denoising individual points independently [23, 24]. Our methodology aligns with the latter, Diffusion Transformer [25) architecture to de-noise pointwise displacements.
adapt the
Deformable Object Manipulation: Despite its prevalence in the real world, deformable manip- ulation remains challenge in robotics due to the complexity of its underlying dynamics and state spaces. Early works have demonstrated success on deformable tasks such as cloth and bedding

---

manipulation [26, 27, 28, 29, 30, 31, 32], rope manipulation [33], dough rolling [34, 35], and bag opening [36, 37]. These methods, however, often rely on pre-defined features (e.g. corner or wrinkle detection) [38, 32, 30], manually designed action primitives (e.g. pick-and-place, fold, or fling) [27, 28, 26, 38, 33, 31, 29, 34), or simplifying assumptions about the scene (e.g. a 2D tabletop environment) [31, 37, 36, 32. 35], limiting their generalization to a broad range of deformable tasks. More recently, end-to-end visuomotor policies [6, 39, 5, 5,40] have shown some capacity learn non- rigid manipulation tasks. However, our experiments show that one such method (DP3 (39]) fails to generalize to novel objects or unseen object configurations, whereas our approach of non-rigid rela- tive placement shows significantly improved generalization abilities.
3 Problem Statement
Relative Placement for Rigid Objects: In this paper, we study "relative placement" as a general framework that encapsulates many robotics tasks (e.g. placing a mug on a rack, or hanging a cloth on hanger). Given two objects A and B, the goal of relative placement task is to manipulate object A (the "action" object) into some position relative to object B (the "anchor" object).
We will briefly review relative placement tasks for rigid objects as defined in prior work [1]: for objects A and B, let point clouds PA and P denote their respective object geometries in desired goal configuration. Suppose, for example, that object A a mug, object B is mug rack, and (Pa, Pis) are the point clouds of these objects when the mug is hanging on the rack. For a relative place- ment task, if both objects are transformed by the same SE(3) transformation T. then the resulting configuration should also be considered successful task completion; for example, if the mug and rack are transformed together, then the mug is still hanging on the rack. Formally, if (PA, Pg) de- note the point clouds of objects A and B in some arbitrary configuration, we can define whether this configuration represents a successful relative placement as:
RelPlace P. PB)) = SUCCESS
JT E SE(3) s.t. PA T PA and P8 = T P
(1)
Then, given observed object point clouds PA and Ps. the goal of a rigid relative placement task [1] to learn a function A Ps) = TAB that predicts a rigid transformation of object such that RelPlace(TAB: PA.PE SUCCESS. The transform T AB is referred as the "cross-pose" [1] since it captures the pose relationship between objects A and B. Using motion planning, the robot can then move object A into new pose determined by TAB to complete the relative placement task, e.g. to place the mug onto the rack [1].
Doos--Dipllacemet for Deformables: In the deformable case, however, this formulation ill- defined. For a task with deformable object A. there is no guarantee that there exists a rigid transfor- mation AAS that can transform an observed point cloud PA into goal configuration PA such that RelPlace (T AB P8) = SUCCESS, as object A may need undergo non-rigid transformation to achieve its goal configuration. Consider, for example, the task of hanging towel towel rack the deformations undergone by the towel as it folds and drapes over the hanger cannot be captured by a rigid transformation.
Instead, to model object deformations, we learn a dense transformation function that predicts where each point in object A must move. Namely, given an initial observation of object A point cloud PA [Tn, , .NN } with x; E R33 we aim learn function g AP Ps) AX, where A.X (Aro. A with Az; E R², such that ReIPlace(PAA + AX, PB) SUCCESS. Intuitively, A is set of dense displacements such that x;; + Ax; brings each point æ; to the goal configuration relative to Pg. As such, we refer to AX as the "cross-diisplacemen"" of objects A and B, analogous to the "cross-pose" defined previously.
Goal Multimodality:
For some tasks, there may be more than one way to achieve a goal configu- ration. For example, for the task of hanging a towel on rack, if object B consists of multiple towel racks, then hanging the towel any of the racks should yield valid goal configuration. To this end, we can define distributional relative placement task: for given point cloud P of object B,
3

---

let mP)))) be set of point clouds for object A such that for all PA E m(P)), the configuration pair (P PB) completes the task. We can then explicitly define success for a distributional relative placement task as:
RelPlaced (P Ps) SUCCESS
TE SE(3), PA mPP)))
s.t. PA = PA and Ps T-
(2)
This definition extends the idea of distributional relative placement defined in prior work [13] to a continuous rather than discrete set of possible goal configurations. To achieve a multi- modal deformable relative placement task, we aim to learn distribution over cross-displacements g(P A, Ps) p(AX) from which can sample a candidate cross-displacement AX p(AX).
Assumptions: For our method, we assume access to segmentations for action object A and an- chor object B. During training, we assume access to a set of M demonstration configurations that PB where (P... PA s PB..) respec- tively denote the initial action object point cloud, goal action object point cloud, and goal anchor object point cloud in demonstration We also assume access correspondences between the initial PA. and the goal P PAi point clouds; these correspondences required to compute the ground-truth cross-displacement A PA PA used to supervise our model as explained below.
complete the task ((PA,,, , PAA.: P Pi..)...., (PA.M; ;; P AAM
4 Method
Given point clouds (A Pι) of objects A and B in an arbitrary configuration, our goal learn function that predicts distribution over dissaasspla A. p(AX). From this distribution, we can sample
displaasplla AX p(AX) transform PA into successful goal configuration PA P. + AX relative to object B.
To solve this distributional non-rigid relative placement problem, we leverage diffusion models [14, 15], a class of generative models that rely on iterative noising and de-noising. More specifically, our method builds on Improved Denoising Diffusion Probabilistic Models [41]. Given some Gaussian noise 2T (noised from the forward process [14)), train model de-noise dr by learning the reverse diffusion process [14]: po(x,-1 xx) = NZz--i re- re-parameterize the mean (1xz and the covariance Ex(x,) using noise prediction (2,)) and interpolation vector væ(x;), respectively, and supervise with a hybrid loss function that combines the standard noise prediction error with a variational lower bound loss; for further details, we defer to Nichol and Dhariwal [41].
(( (xx)), >a (zx)). Following [41], we
4.1 Point Cloud Generation for Relative Placement
Training: Given a sample demonstration (P PA Pi)), where PA is the initial action object point
cloud (z.A.o.... EA.N), ). PAA is the goal action object point cloud {== A........... , 'AAN and P is the goal anchor object point cloud, we train our model to de-noise a set of per-point displacements AX,
with the ground-truth displacements defined as A PA - PA = {=A, 2 x AAAN-
T..N]. Following Ho et al. [15], we sample partially noised displacements AX₁:
AX, = Va,AXo + /i -
(3)
where AX, = Ax [0, ~ (N, and ā, is a function of the noise schedule. We then supervise our model's noise co( (o,,,, P... P. f) and interpolation vector by (AX,, PA, Ps. t) predictions using the hybrid loss function from Nichol and Dhariwal [41]. Note that the model is conditioned on the anchor P well the initial action point cloud P.A. which refer below as the "action context." For architecture and training specifics, see Appendix B.
Inference: At inference, given some object configuration PB), we initialize a set of per per-point displacements as Gaussian noise: AX, I),, These displacements are iteratively de-noised using our model's outputs: AX,-1 ~ A (AX,- μog Ee), with μı and Eg re-parameterized as in Nichol and Dhariwal [41]. The final de-noised displacements A. then used to transform the points of PA into predicted placement PA A relative to Pg.

---

(axx Pa.Pal
A.T
Jani
Point Cloud
Diffusion Transformer
P7
Figure 2: (Left). During inference, randomly sampled displacements AX, (0, I) are de-noised conditioned on action (P and anchor (P&) features; the final AX is predicted to displace the action into goal configuration. (Right). Our modified DiT [25] architecture combines self-attention and cross-attention for object-centric and scene-level reasoning.
Object-Speciine Frames: To guarantee translation-invarian in our model, we process input point clouds (PA,, PB) in object-specific frames. Namely, we center the action context PA in the action frame (P's PA- PP)) and the anchor Pg in the anchor frame (P'g Ps Pι) by subtracting their respective point cloud means PA and Pg. Accordingly, our model is conditioned on (P P'g) instead of (PA, Ps). During training, we further center the ground-truth goal action point cloud in the anchor frame (P* PA -B)) and modify the ground-truti displacements to AX* easily transform our predictions back into the world frame by inverting
-P'--
During inference, these translations. In our experiments, we show that this use of object-specific frames is crucial for robust out-of-distribution generalization.
4.2 Diffusion Transformer for Point Cloud Generation
We adapt the Diffusion Transformer (DiT) [25] architecture for our model, as shown in Figure 2. At each diffusion timestep, our model takes as input the noised displacements AX,, the action context PA the anchor Ps. and the timestep t. Using MLP encoders, we first compute per-point displacement features fax, per-point action context features f& and per-point anchor features f from AX,, PA. and Ps. respectively. MLP weights are shared within sets of features. The action context features and displacement features are then concatenated into (frA fAz) as input modified DiT model alongside anchor features L (see Figure 2, left).
Within a DiT block, self-attention is applied to the combined action features fam, to aggregate information across the entire action point cloud and facilitate coordinated displacement predictions. Cross-attention is then applied to these features with the anchor features f allowing for scene-level global reasoning between the action and anchor objects. This repeated for N blocks, before the network outputs noise cø and interpolation vector vo prediction, described above.
We refer to the above approach as the *Cross-Displacement (CD)" variant of our TAX3D architec- ture, since we directly predict the cross-displacement AX. We also propose a "Cross-Poine (CP)" variant which we directly encode and diffuse over the positions of the predicted goal point cloud PA = PA + AX instead; we find that both variants perform similarly in our experiments. Follow- ing Peebles and Xie [25], incorporate timestep conditioning using adaptive layer normalization (adaLN) blocks. For permutation equivariance, we do not incorporate any positional encoding.
Experiments
To the best of our knowledge, quantitative benchmarks do not exist for the setting of de- formable object relative placement. As such, to evaluate our approach, we design novel experimen- tal benchmark building on DEDO: Dynamic Environments with Deformable Objects [7], suite of simulation environments for deformable manipulation. We experiment two cloth-hangin tasks (Figure 3): HangProcCloth, in which cloth must be hung through one of its holes, and HangBag,
Setup:
5

---

Figure 3: TAX3D generalizes to diverse cloths and anchor positions (top); we also visualize the cor- responding goal predictions (middle) and successful rollouts (bottom) after releasing the cloth. The two rightmost columns are HangBag configurations all others are HangProcCloti configurations.
in which a bag must be hung on one its handles. For environment and demonstration details, see Appendix We also evaluate our method in the real world, described further below.
Evaluation Metrics: We evaluate approach using two metries: point prediction error and downstream policy performance. To measure point prediction error, we use point-wise root-mean- square-error (RMSE) to compute the distance between a prediction P P.A and a ground truth goal configuration PA- To evaluate distributional placements for multimodal experiments, we also report Coverage RMSE, in which we compute the minimum distance to set sampled predictions (P for each ground truth goal A and Precision RMSE, in which we compute the minimum distance the set of ground truth goals (Pa,.) for each sampled prediction P PA.j- To evaluate downstream policy performance, we use primitive goal-conditioned policy that directly converts a predicted goal point cloud PA into position control inputs and report the cloth hanging Success Rate. Further details regarding evaluation can be found in Appendix C.
Ablations and Baselines: We evaluate the two variants of our method described in Section 4.2 "Cross- CCross-Displacemenn" (CD) and "Cross-Poine Point (CP) against several architectural ablations:
(1) Scene Displacement/Point (SD/SP): a variant of our CD/CP architectures without object- centric reasoning. The action and anchor point clouds are combined into single scene point cloud Ps. and the DiT blocks are implemented with self-attention only.
(2) Cross Displacement/Point World Frame (CD-W/CP-W) a variant of our CD/CP chitectures without object-specific frames: P. and P& are not mean-centered.
(3) Cross Displacement/Poin' No Action Context (CD-NAC/CP-NAC): variant of our CD/CP architectures without action context features.. PA is not encoded, but cross-attention between displacement features faz and anchor features f is retained.
We also compare our method to 3D Diffusion Policy (DP3) [39], a recent end-to-end visuomotor policy that has achieved state-of-the-art manipulation results using point cloud inputs.
Generalization to Unseen Configurations and Geometries: Experiments are conducted on two versions of the HangProcCloth task (unimodal, in which the cloth has one hole, and multimodal, which the cloth has two holes) as well as on the HangBag task. Important comparisons are shown in Tables 1, 3, and 4, respectively, with full results in Appendix D. Training demonstrations are gen- erated under random anchor poses (64 demonstrations for HangProcCloth and for HangBag), and models evaluated
of configurations (40 trials each): Unseen, which contains novel anchor poses from the training distribution, and Unseen -Outtif-Distriuution) which anchor poses. For both HangProeCloth tasks, cloth and hole shape are
tains

---

RMSE (4)
Success Rate (1)
Train
Unseen
Unseen
Train
Unseen
Unseen
(OOD)
(OOD)
SD
0.127
0.805
4.246
0.27
0.20
0.00
SP
0.095
0.779
3,955
0.45
0.48
0.08
CD-W
0.099
0,464
12.464
0.95
0.88
0.00
CP-W
0.085
0.511
19.923
0.95
0.85
0.00
CD-NAC
2.233
2.243
2.326
0,16
0.28
0.10
CP-NAC
3,987
3,934
3,923
0,02
0.00
0.03
TAX3D-CD (Ours)
0.052
0.405
0.395
0,94
0.90
0.85
TAX3D-CP (Ours)
0.051
0.390
0.422
0.93
0.95
0.80
DP3
0.98
0.38
0.03
Table I: soaeProoClothhumimddaal Random Cloth Geometry (1-Hole). We do not report RMSEs for DP3 since it end-to-end policy and does not predict point clouds.
randomized across demonstrations, with Unseen and Unseen (OOD) consisting only of unseen cloth geometries. For the HangBag task, only a single fixed cloth geometry is used.
All ablations (Table 1) fail on out-of-distributiio an- chor poses, demonstrating that method's com- bination of object-level reasoning in object-specific frames is crucial for generalization to novel scene configurations. As the point prediction errors indi- cate, NAC (No Action Context) ablations are unable produce meaningful cloth geometries (Appendix D) due to the lack of action context crucially, with- out combined action features (frt faæ), the net- work cannot maintain correspondences between dis- placements Ar; and action points
Success Rate (t)
Unseen
Unseen (OOD)
TAX3D-CD
1.00
0.98
TAX3D-CP
1.00
0.98
DP3
0.93
0.00
Table 2: HangProcCloth-simmlle Fixed Cloth Geometry
We also find that DP3 fails to generalize both to novel scene config- urations and to novel object instances, despite fitting well to training demonstrations. To understand why, conduct an additional experi- ment with fixed cloth geometry [IangProcCloth-simppee Table 2), in which DP3 achieves decent performance Unseen (but in-distribution) anchor poses. This indicates that the poor performance on Unseen in Caw-rrocooot--minooo
is result of DP3's inability to adapt to vari- ations in the cloth geometry. TAX3D, on the other hand, generalizes well to both scene and object variations, maintaining significantly supe- rior performance across all HangProcCloth and HangBag experiments (Tables 3, 4) in comparison to DP3.
Figure 4: Multimodal TAX3D predictions (left), with successful rollouts (right).
Multimodal Goal Prediction: Our diffusion-basse architecture nat- urally accommodates multimodal settings (Figure 4). To explicitly measure this, addi- tionally compare against regression baselines (Regression Displacement/Point (RD/RP)) for SatpPrrrocoottmmultimeee
To do so, we remove the timestep conditioning from the archi- tecture and train the model to directly predict PA via a regression loss (MSE). As Table indicates, the regression models lead to poor performance on the multimodal task variant of a cloth with two holes. In contrast, our method continues to achieve good performance on this task; the low coverage and precision RMSEs indicate that it can predict both modes, while avoiding the mode-averaging tendencies of regressive inference.
Real World Experiments: We also qualitatively demonstrate our approach in a real-world cloth- hanging setup. Human demonstration videos captured using single Azure Kinect camera. Seg- mentations and correspondences are then obtained using prompep Segment Anything [42]

---

Coverage RMSE (4)
Precision RMSE ())
Success Rate (1)
Unseen
Unseen
Unseen
Unseen
Unseen
Unseen
(OOD)
(OOD)
(OOD)
RD
2.987
3.057
1.087
1.114
0.60
0.50
RP
1,417
1.432
1.192
1.154
0.65
0.48
TAX3D-CD (Ours)
0,457
0.543
0.403
0.558
0.98
0.73
TAX3D-CP (Ours)
0.453
0.540
0.495
0.631
0.85
0.70
DP3
0.45
0.00
Table Hang@rooClooth-multiinoooo Random Cloth Geometry (2-Hole)
RMSE (L)
Success Rate (1)
Train
Unseen
Unseen
Train
Unseen
Unseen
(OOD)
(OOD)
TAX3D-CD (Ours)
0.183
0.204 0.192
0.245 0.233
0.94
0.83
0.78
TAX3D-CP (Ours)
0.173
0.94
0.93
0.78
DP3
0.98
0.38
0.03
Table 4: HangBag: Fixed Cloth Geometry
Figure 5: Real world results. TAX3D succeeds under varying anchor poses, varying peg placements (left, middle-left), and can model multimodal placements with multiple pegs (middle-right, right).
and SpatialTracker [43], respectively. We train our model on 10 demonstrations, containing multi- modal placements and various anchor poses. To perform robot actions, we use position control to track the predicted goal point cloud. As shown Figure 5, our method can complete the task under novel anchor poses and anchor geometries, while accommodating multimodal goal predictions.
Conclusion
In this paper, present novel framework perform relative placement for non-rigid manipulation tasks. We formulate the task of "cross-displacemenn" prediction to handle relative placement for arbitrary object deformations, and we show that our dense diffusion architecture can learn cross- displacements on multiple cloth hanging tasks in simulation and in the real world. Our experiments demonstrate our approach's 's ability to generalize to out-of-distribution scene configurations, unseen object instances, and multimodal placements for robust deformable manipulation.
Limitations:
There are several limitations our method, which we leave for future work:
1. Segmentations. Our method requires segmented action and anchor point clouds. Our cur- rent approach for obtaining such segmentations prount-rommped Segment Anything [42]) requires human involvement, limiting scalability. This can potentially be addressed using language-conditioned segmentation methods [44] to automate demonstration labeling.
2. Open-loop control. Our method currently relies on open-loop position control the pre- dicted goal for robot execution, which limits the ability perform complex placement tasks. This potentially be addressed by incorporating TAX3D goal predictions into goal-conditioned manipulation policy.

---

Acknowledgments
This material is based upon work supported by the Toyota Research Institute, the National Sci- ence Foundation under NSF CAREER Grant No. IIS-2046491, and NIST under Grant No. 70NANB23H178. We are grateful to Prof. Shubham Tulsiani for his valuable feedback and discus- sion at various stages of the project, and toLifan Yu for her assistance with real-world experiments.
References
[1] C. Pan, B. Okorn, H. Zhang, B. Eisner, and D. Held. Tax-pose: Task-specific cross-pose esti- mation for robot manipulation. In Conference on Robot Learning, pages 1783-1792. PMLR, 2023.
[2] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se (3)-eqiiarriant object representations for manipulation. In 2022 International Conference Robotics and Automation (ICRA), pages 6394-6400. IEEE, 2022.
[3] P. R. Florence, L.. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint arXiv: 180..0876, 2018.
[4] A. Simeonov, A. Goyal, L. Manuelli, Y.-. Lin, A. Sarmiento, A. R. Garcia, P. Agrawal, and
D. Fox. Shelving, stacking, hanging: Relational pose diffusion for multi-modal rearrangement. In Conference on Robot Learning, pages 2030-2060 PMLR, 2023.
[5] J. Yang, C. Deng. J. Wu, R. Antonova, L. Guibas, and J. Bohg. Equivact: Sim (q)-eqviva visuomotor policies beyond rigid object manipulation. arXiv preprint arXiv:2310.16033 2023.
[6] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[7] R. Antonova, P. Shi, H. Yin, Z. Weng, and D. K. Jensfelt. Dynamic environments with deformable objects. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[8] B. Eisner, Yang, Davchev, M. Vecerik, Scholz, and D. Held. Deep se(3) selui-equivariant geometric reasoning for precise placement tasks, 2024.
[9] Y.-C. Lin, P. Florence, A. Zeng, Barron, Du, W.-C. Ma, Simeonov,
Garcia, and P. Isola. Mira: Mental imagery for robotic affordances. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1916-1927. PMLR, 14-18 Dec 2023. URL https: // //ppoceedings mlr press/v205/1in225 html.
[10] J. Seo, N. P. S. Prakash, X. Zhang, C. Wang, J. Choi, M. Tomizuka, and R. Horowitz. Contact- rich se(u)-equivariant robot manipulation task learning via geometric impedance control. IEEE Robotics and Automation Letters, 952) 1515, Feb. 2024. ISSN 2377-3774. 1oi:10.1109/ 1ra.2023.3346745 URL http:://tx....// doi 1019/.0...09/LA... 2....3342744 3346748.
[11] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, A. Wahid, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation, 2022.
[12] A. Simeonov, Y. Du, L. Yen-Chen A. Rodriguez, L. P. Kaelbling, T. Lozano-Perez, and Agrawal. Se(3) (equ)--quiva relational rearrangement with neural descriptor fields, 2022.
[13] Wang, O. Donca, and D. Held. Learning distributional demonstration spaces for task-specific cross cross-pose estimation. IEEE International Conference Robotics and Automation (ICRA), 2024, 2024.

---

[14] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learn- ing using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-22655 PMLR, 2015.
[15] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840- 6851, 2020.
[16] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image syn- thesis with latent diffusion models, 2022.
[17] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis, 2021.
[18] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents, 2022.
[19] C. Saharia, W. Chan, S. Saxena, L., J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi. Photorealistic -ikttto-mmae diffusion models with deep language understanding, 2022.
[20] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
[21] L. Zhou, Y. Du, and J. Wu. 3d shape generation and completion through point-voxel diffusion, 2021.
[22] X. Zeng, A. Vahdat, F. Williams, Z. Gojcic, O. Litany, S. Fidler, and K. Kreis. Lion: Latent point diffusion models for 3d shape generation. Conference on Neural Information Processing Systems (NeurIPS), 2022.
[23] S. Luo and W. Hu. Diffusion probabilistic models for 3d point cloud generation. Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[24] G. Nakayama, M. Uy, J. Huang, Hu, K. Li, and L. Guibas. Difffacto: Controllable part- based 3d point cloud generation with cross diffusion. International Conference on Computer Vision (ICCV), 2023.
[25] W. Peebles and S. Xie. Scalable diffusion models with transformers. International Conference on Computer Vision (ICCV), 2023.
[26] X. Lin, Y. Wang. Z. Huang, and D. Held. Learning visible connectivity dynamics for cloth smoothing, 2022. URL wwtphhtts:///rat http//aawwv...g/abaeaaas/ess
org/abs/////// 10389.
[27] H. Ha and Song. Flinghot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding, 2021. URL httpps//aarxw...rrg///////////// 03655.
[28] Z. Xu, C. Chi, B. Burchfiel, E. Cousineau, S. Feng, and S. Song. Dextairity: Deformable manipulation can be a breeze, 2022. URI. https: / arxiv. wwtp:/s////aaa////eeee 011997
[29] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to manipulate deformable objects without demonstrations, 2020. URL titts:///arivv http:://wwwvvvvwvaaaeeeeeee wtt.:org///aaaaaoooaa 1bt1//0..1991111.9..... 13439.
[30] J. Shepar,, M. Cusumano- Cusumano-Townnr, J. Lei, and P. Abbeel. Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding. In 2010 IEEE International Conference on Robotics and Automation, pages 2308-2201 2015, 2010. doi:
10.1009RRB00T220005599999
[31] T. Weng, S. Bajracharya, Y. Wang, K. Agrawal, and D. Held. Fabricflownet: Bimanual cloth manipulation with flow-based policy, 2022. URL https: //wrxiva.ivvv//// org/abs/2111.00 05623.
10

---

[32] K. Sun, G. Aragon-Camarase P. Cockshott, S. Rogers, and J. Siebert. A haases ap- proach for flattening wrinkled clothes. volume 8069, 08 2013. ISBN 978-3-662-43644--8 doi: 10077997--3-62--4797
[33] D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng. Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks, 2023. URL https: btttpa///ariivi http:///xxvvvoroorro org/ (wrrg/abs/2115 03385.
[34] J.-T. Kim, F. Ruggiero, V. Lippiello, and B. Siciliano. Planning Framework for Robotic Pizza Dough Stretching with Rolling Pin, pages 229-253. 03 2022. ISBN 978-3-030-93289-3.
1e111...109997 (007/978-330030-3299--999 93290-9.9.
[35] C. Qi, X. Lin, and D. Held. Learning closed-loop dough manipulation using differentiable reset module, 2022. URL https://arxii: org/ absa 2207 046388.
[36] L. Y. Chen, B. Shi, D. Seita, R. Cheng, T. Kollar, D. Held, and K. Goldberg, Autobag: Learning to open plastic bags and insert objects, 2023. URL https://arxix: wwt.:oogg//a/a/eeeeooooo 17217.
[37] A. Bahety, S. Jain, H. Ha, N. Hager, B. Burchfiel, E. Cousineau, S. Feng, and S. Song, Bag all you need: Learning a generalizable bagging strategy for heterogeneous objects, 2023, URL https: / /arxiv.xivvv///.v///aa//aaaa . org/ 2abb:/221.0000 09997.
[38] D. Seita, N. Jamali, M. Laskey, A. K. Tanwani, R. Berenstein, P. Baskaran, S. Iba, J. Canny, and K. Goldberg, Deep transfer learning of pick points on fabric for robot bed-making, 2019. URL httttttp//aarrre
org/abs// wwtaaa///////////e 09810.
[39] Y. Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations, 2024. URL htttttt//axiii..oo abs/2403.03955
[40] Yang, ang Cao, C. Deng, Antonova, S. Song, J.d Bohg. Equibot: Simi3)-equivariant diffusion policy for generalizable and data efficient learning, 2024. URL httph::t///aee org/ hrg/abs//e abb//407 01479.
[41] A. Q. Nichol and Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional conference on machine learning, pages 8162-8171. PMLR, 2021.
[42] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything, 2023. URL https:
// farxiv.or//aiii wtwp///////////a///e/e////// /304. 02643.
[43] Y. Xiao, Q. Wang. S. Zhang. N. Xue, S. Peng, Y. Shen, and X. Zhou. Spatialtracker: Tracking any 2d pixels in 3d space, 2024. URL htttp:///axxee org/ abs/2404. 2404.04319.
[44] Ren, Liu, A. Zeng. Lin, Li, H. Cao, Chen, X. Huang, Y. Chen, Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arX1v:2401.1415,, 2024.
[45] E. Coumans and Y. Bai. Pybullet, a python module for physics simulation for games, robotics. and machine learning, 2016-2020. URI. htttt:://yybuulet..oo.

---

Appendix
Table of Contents
A DEDO Environment Details
13
A.1 HangProcCloth Task Definition
13
HangProcCloth Cloth Generation
A.3 HangBag Task Definition
14
A.4 Cloth Control
15
A.5 Episode Rollout
15
A.6 Demonstration Generation
16
B Training Details
17
Model Architecture
Training Pre- Processing & Hyperparameters
C Evaluation Metrics
18
D Experiments
18
D.1 HangProcCloth-simpple Results
19
D.2 HangProcollot--mninotall Results
19
langrrooolot-mmlliindaalll Results
D.4 HangBag Results
................................
20
D.5 Additional Visualizations
20
12

---

A DEDO Environment Details
DEDO: Dynamic Environments with Deformable Object [7] a suite of simulation en- vironments (hanging a bag, dressing a mannequin, etc.) involving highly deformable, topologically non-trivial objects. The environments are built on the PyBullet physics engine [45].
Floating Grippers
Cloth
'Hanger'
Loop Vertices
'Tall Rod'
Initial State
Post-Manige Maaaaaaappllaiinn
Post-Reieer Release
Figure 6: Sample demonstration of the HangProcCloth task.
A.1 HangProcCloth Task Definition
For most of our experiments, we focus on the HangProcCloth task (Figure 6), in which procedu- rally generated cloth must be placed on a hanger. More specifically, the cloth is generated to contain a hole its topology to successfully complete the task, the vertical part of the hanger should be aligned through the hole.
The hanger (anchor) is loaded into the PyBullet engine pre-defined rigid body, and contains two components: 'tall rod', and the 'hanger' itself. While randomize the anchor pose throughout our experiments, this geometry remains fixed. The goal of the task is explicitly formulated in the environment as the center the 'hanger' component (Figure 6) while this goal definition is not passed as input to our models, it is used later by our success metric for evaluation.
A.2 HangProcCloth Cloth Generation
Following DEDO's s implementation, every cloth in our experiments procedurally generated as rectangular mesh, and can be represented using the following parameters:
node_density
25
The amount of vertices to initialize the cloth mesh with. Every cloth is initialized an evenly spaced node_density x node seleeddentttyyeeee grid (25 x25 625 vertices for all of our cloths). Vertices are then removed during the hole generation process.
width
[0.8, 1.2]
The width of the cloth.
height
[0.8, 1.2]
The height of the cloth.
num holes
(1..2
The number of holes in the cloth.
holes
See A.2.1
See A.2.1
13

---

A.2.1 HangProcCloth. Hole Generation
Holes are created by removing mesh vertices, All generated holes are rectangular such, they can be represented topologically with respect to the procedurally generated cloth by their bottom-left and top-right corners. Accordingly, the holes parameter is a list, where each element corresponding to specific hole in the cloth is a dictionary with elements:
x0
The x vertex coordinate of the bottom-left corner of the hole.
yo
The vertex coordinate of the bottom-left corner of the hole.
x1
Similar to x0, for the top-right comer.
y1 Similar yo, for the top-right corner.
For reference, the single-hole cloth used in HangProcCloth-simnpe experiment defined
2
"node_ prode_density". 25,
"width": :: 1.0,
"height" ; 1.0,
"num_holes" 1,
"holes":
{ xx00: 8, "yo": 9, "x1": 16, "y1": 13)
8
]
In general, holes are randomly generated under the following constraints:
x x.ngee
(2, node density 2)
The range of possible values for x0.
y-range
(2, node 2)
The range of possible values for y0.
width
(5, 7)
The range of possible values for wh. such that
x0 + wh.
height.range
(5, 7)
The range of possible values for wh. such that x1 x0 wh.
More precisely, when generating holes, x0 and yo are first sampled based on x x.range and y yrrngge, respectively x1 and y1 are then sampled based on width_range and height_range To ensure that the resulting cloth geometry is valid topologically, DEDO generates cloths using Monte Carlo method, only returning valid holes if they pass boundary check (all vertices lie within the cloth boundary) and an overlap check (different holes do overlap). For further implementation details, refer to the DEDO codebase [7).
Since holes are generated by directly manipulating the cloth mesh they can also be represented as deformable loops, defined by a set of "loop vertices" (Figure 6) while information about these vertices is not passed as input to our models, is used later by our success metric for evaluation.
A.3 HangBag Task Definition
We also perform additional experiment the HangBag task (Figure 7), in which deformable bag with two handles must be placed hanger. To successfully complete the task, at least of the handles must lie on the handle.
Similar to HangProcC1oth, the hanger (anchor) a pre-defined rigid body for our experiments, we randomize the anchor pose, but keep the geometry fixed. Unlike HangProcCloth, however, the cloth is loaded from a pre-defined mesh rather than procedurally generated. As such, we do randomize the cloth geometry for this task, leaving such experiments for future work (notably, the DEDO environment does provide multiple meshes for the bag).
14

---

'Hanger'
God Locstinn
Ficating Grippers
Cloth
'Tall Rod'
Laop Veriias
Initial Stale
Moap-Maanpulatioon
Post-Release
Figure 7: Sample demonstration of the HangBag task.
A.4 Cloth Control
The HangProcCloth and HangBag environments do not model robot grasp- instead, the cloth is manipulated by applying force controls to floating "grippers" attached to pre-defined points on the cloth (Figure 6,7). The grippers themselves are zero-mass and collision free.
Pseudo-expert Policy: To generate demonstrations, we hard-code a pseudo-expert policy with access to privileged environment information. At the initial state of the scene, the policy computes a target position for each gripper using the distance from the centroid of the loop vertices to the goal location in the hanger. this target, we apply the same transformation that we apply to the anchor to randomize its pose this ensures that the cloth is rotated appropriately align with the hanger. each time step, also add small correction to the target position based on the distance between the current centroid of the loop vertices and the goal position this allows the pseudo-expert policy to adapt to small deformations in the cloth geometry, which we find meaningfully improves the success rate. For control, we pass the target positions for each gripper as inputs to a custom pypportionaldderiaane to target velocity 0) with velocity and position gains of 50 and a maximum force? 5.
cloth has multiple holes, a single hole is selected. When computing
controller (in addition
Evaluation Policy: To evaluate our models, we implement a separate evaluation policy without access to privileged environment information (e.g. goal location, deformable loop vertices). At the initial state of the scene, we run TAX3D on the full point cloud of the cloth", and obtain the predicted position in the world frame of the two grippers (which assume attached to known points of the cloth). These target positions are then passed as inputs (in addition to target velocity of zero) to custom proportional-derivtive controller, with the same gains as the pseudo-expert policy.
A.5 Episode Rollout
A.5.1
Rollout Phases
Fach episode rollout consists of two phases (Figure 6): manipulation phase, in which the grippers receive force control inputs at each time step to manipulate the cloth, and a release phase, in which
DEDO refers to these grippers as "anchors." T We refrain from this terminology since "anchor" denotes an entirely different object for our purposes.
Followwig the DEDO implementation, this is not overall maximum force magnitude is the maximum magnitude of the force along the x- y- and z-axes.
Note that this is different from our training procedure B.2), where we downsample cloth point clouds to 512 points.
15

---

the grippers "release" the cloth and allow it to fall. The release phase is fixed at 500 simulation steps, whereas the manipulation phase has a variable episode length depending on the setting (with each environment step corresponding to simulation steps).
If the task is completed successfully, the cloth should be supported by the rigid anchor after the release phase. However, because are learning goal-prediction module condition policy's control outputs, we use the post-manipulation, pre-release state of the cloth to label ground truth demonstrations.
A.5.2 Success Metric
To robustly determine whether or not the task has been successfully completed, we implement our own success metric consisting two components:
1. Centroid Check: binary metric that checks if the centroid of the deformable loop vertices
is within threshold distance of the goal location.
2. Polygon Check: binary metric that projects the loop vertices and goal location onto the ryy-lane, and then checks if the projected goal point lies on the interior of the polygon defined by the projected loop vertices. This is an intuitive heuristic that checks whether or not the hole "wraps" around the vertical rod of the hanger.
If the cloth has multiple holes, these metrics computed individually for each hole the task is considered successful if both are true for at least one hole.
For the HangProcCloth task, the success metric consists of a centroid check (with threshold 1.3) pre-release and a polygon check post-release. For the HangBag task, the success metric conists of two centroid checks, both with threshold 1.4, pre- and post-release we do not use the polygon check for the HangBag task, we found that it produces larger number of false negatives.
A.6
Demonstration Generation
A.6.1
Randomizing Scene Configuration
For all HangProcCloth experiments, the objects in the scene are initialized to the following pose, shown in Figure 6:
position (ryz)
orientation (Euler)
cloth hanger
(0, 5, 8)
(- - 0,
(0, 0, 8)
(0, 0, 0)
tall rod
(0,
0,
Table
Initial HangProcCloth configuration
For all HangBag experiments, the objects in the scene are initialized to the following pose instead (shown in Figure 7):
position (zyz)
orientation (Euler)
cloth
(0, 8, 2)
[ 0, 0)
hanger
(0, 1.28, 9)
0,00, 7
tall rod
(0, 0, 5)
(5,0,
Table 6: Initial HangBag configuration
16

---

To randomize scene configuration, the anchor is then transformed with randomly sampled transla- tion, and a randomly sampled rotation about the z-axis.
Unseen
Unseen (OOD)
e-trranslatonn
(-5.5)
(-10, -5) U (5, 10)
y-translation
(0,-10)
(0, (0,-
e-translation
(1,5) 5)
2-rotation
(- 7
(-3 x
All transformations are sampled uniformly at random from their respective ranges, with one small caveat: z-translations are chosen such that their signs match the sign of the sampled rotation. That is, the z-rotation sampled to be non-negative, then z-translations are only sampled from the non-negative subset of the corresponding range. This ensures that the anchor always "faces" the cloth, such that the cloth need not undergo significant rotations for successful placement.
As for the point clouds themselves, we obtain P as a partial point cloud from an RGB-D render of the initial state of the environment (since the anchor is static). To guarantee correspondences, PA and PA are directly extracted from the mesh vertices of the cloth at its initial and post-manipulatin states, respectively.
A.6.2 Experiment Datasets
As reminder, cloth geometry (including holes) is randomized by following the parameter ranges and procedures described in 1. The datasets for each experiment are generated as follows, where each tuple entry corresponds to the (Train, Unseen, Unseen (OOD)) settings, respectively:
cloths
# holes per cloth
# total demonstrations
HangProoClooth-ssmmlle Hang@PoocCooth-niinooe
(1, I, (64, 40, 40) (32, 20, 20) (1, 1, I)
(1, I)
(16, 40, 40)
(1, I, 1)
(64, 40, 40)
HangProcCloth htt.rrooooottmmmmmmmmmmm...aea
(2, 2,
(64, 40, 40)
HangBag
(1, 1)
(16, 40, 40)
For HangProcClotthmmltiiiee such, there are half many unique cloths there total demonstrations. For all demonstrations across all experiments, the anchor pose is randomized as described in A.6.1.
we generate a successful demonstration for each hole per cloth
B
Training Details
B.1 Model Architecture
(asa Pa)
A.Y
fanus
ifA fonul
Uth fae
Point Cloud Diffesion Transformer
Ps
Figure 8: TAX3D model architecture. (Left). During inference, randomly sampled displacements
AX, ~ I0, are de-noised conditioned on action (P and anchor (Ps) features; the final AN, is predicted displace the action into goal configuratinn. (Right). Our modified DiT [25] architecture combines self-attention and cross-attention for object-centric and scene-level reasoning.
As discussed in 4.2, modify the standard DiT block [25] include additional cross-attention
head 8. For all of our experiments, train the same architecture:
17

---

depth num heads hidden size
5
of DiT blocks heads per block
128
hidden size per block
These settings (namely, num heads and hi hidden size) are applied identically to the self-attention and cross-attention layers. During training and inference, our model always uses 100 diffusion steps, with a linear noise schedule.
B.2 Training Pre-Processing Hyperparameters
For training, both the action and anchor point clouds are downsampled to 512 points using furthest point sampling". The anchor point cloud is additionally augmented with 2-axis rotations sampled uniformly random from [0, 2T)
All models are trained under the same hyperparameters with AdamW optimization and cosine scheduling with warmup:
learning rate learning rate warmup steps
1 x 10 100
weight decay epochs batch size
1 10-5 20,000
16
C Evaluation Metrics
As discussed in Section 5, our method's modeling of point-wise displacements allows us to directly use root-mean-squart saeaan--uuueedeeeee (RMSE) as a distance metric between predicted and ground truth con- figurations of the cloth. To appropriately evaluate distributional predictions in our setting, we define two evaluation metrics::
1. Coverage RMSE:
For each demonstration with ground truth PA, we sample 20 predic- tions (PA.))- and keep the minimum RMSE. This is aggregated across all demonstrations in the dataset. Intuitively, this metric captures how well a model can produce all of the modes given dataset that is, how well it covers distribution.
2. Precision RMSE: We first collect demonstrations corresponding to specific cloth geom- etry (this is either one demonstration for the unimodal case, two demonstrations for the multimodal case) for some cloth C, this serves as a cloth-specific reference set (P* Jc We then sample 20 predictions conditioned on cloth C, and compute for each prediction P PAj the minimum RMSE to ground truth point clouds the reference set (c This is aggregated across all 80 predictions, and then across all cloths. Intuitively, this metric captures how well model can consistently produce predictions that are close to the dataset configurations that is, how precisely it models distribution.
D Experiments
The following sections contain all of the ablation and baseline comparisons for all experiments. Note that we do report any RMSE metrics for DP3, since it is an end-to-end policy, and does not predict a point cloud.
'During policy evaluation, only anchor point cloud downsampled, as full action point cloud need to obtain target positions for the grippers. 'Both metrics bear strong similarity to the MMD metric, but essentially modified to aggregate different reference sets.
"In the multimodal case, all demonstrations for single cloth are recorded under the same anchor configu- ration such, the world frame RMSEs are consistent across any cloth-specific reference (Pa.))e.
18

---

D.1 HangProcCloth-simpp Results
RMSE (4)
Success Rate (1)
Train
Unseen
Unseen
Train
Unseen
Unseen
(OOD)
(OOD)
SD
0.054
0.741
2.898
0.94
0.98
0.00
SP
0.051
0.172
2.969
0.50
0.58
0.05
CD-W
0.052
0.224
3.680
1.00
1.00
0.00
CP-W
0.044
0.231
3,452
1.00
1.00
0.00
CD-NAC
1.586
1,498
1.741
0.56
0.60
0.53
CP-NAC
3.563
3.573
3.534
0.00
0.00
0.00
TAX3D-CD (Ours)
0.270
0.255
0.498
1.00
1.00
0.98
TAX3D-CP (Ours)
0.298
0.277
0.518
1.00
1.00
0.98
DP3
1.00
0.93
0.00
Table 7: simgsro Fixed Cloth Geometry
D.2
HangProcCLlot--unnnodaa Results
RMSE (1)
Success Rate (1)
Train
Unseen
Unseen (OOD)
Train
Unseen
Unseen
(OOD)
SD
0.127
0.805
4.246
0.27
0.20
0,00
SP
0.095
0.779
3.955
0.45
0.48
0.08
CD-W
0.099
0,464
12.464
0.95
0.88
0.00
CP-W
0.085
0.511
19.923
0.95
0.85
0.00
CD-NAC
2.233
2.243
2.326
0.16
0.28
0.10
CP-NAC
3.987
3.934
3.923
0.02
0.00
0.03
TAX3D-CD (Ours)
0.052
0.405
0.395
0.94
0.90
0.85
TAX3D-CP (Ours)
0.051
0.390
0.422
0.93
0.95
0.80
DP3
0.98
0.38
0.03
Table 8: HangProoclotth-miinddlll Random Cloth Geometry (1-Hole)
D.3
HangProoCloohhmmlliiine Results
RMSE (L)
Success Rate (1)
Train
Unseen
Unseen
Train
Unseen
Unseen
(OOD)
(OOD)
SD
1.048
1.361
3.832
0.36
0.48
0.05
SP
1.028
1,407
3.825
0.53
0.40
0.05
CD-W
1.436
1.535
5,006
0.95
0.63
0.03
CP-W
1,418
1.555
48.760
0.97
0.70
0.08
CD-NAC
2.467
2.560
2.650
0.25
0.10
0.10
CP-NAC
4.148
4.237
4.114
0,00
0.00
0.00
RD
2.978
2.987
3.057
0.53
0.60
0.50
RP
1.360
1,417
1.432
0.47
0.65
0.48
TAX3D-CD (Ours)
1.413
1.532
1.610
0.95
0.98
0.73
TAX3D-CP (Ours)
1.400
1,568
1.608
0.94
0.85
0.70
DP3
0.97
0.45
0.00
Table HangProcCllthhmmllliin Random Cloth Geometry (2-Hole)
19

---

Coverage RMSE (+)
Precision RMSE ())
Train
Unseen
Unseen
Train
Unseen
Unseen
(OOD)
(OOD)
SD
0.075
0.740
3.500
1.114
1.379
4.214
SP
0.061
0.773
3.462
1.093
1.432
4.062
CD-W
0.054
0.426
4.269
0.132
0,477
4.351
CP-W
0.047
0.446
33.868
0.099
0.504
47.372
CD-NAC
1,484
1,741
1,812
1.785
2.011
2.167
CP-NAC
3,593
3.720
3,603
3,770
3,914
3,786
RD
2.978
2.987
3,057
1,296
1.087
1.114
RP
1.360
1.417
1.432
1.305
1.192
1.154
TAX3D-CD (Ours)
0.040
0.457
0.543
0.084
0.403
0.558
TAX3D-CP (Ours)
0.038
0.453
0.540
0.087
0.495
0.631
DP3
Table 10: Hang@PoocC.ochhmmltiinnoooo Random Cloth Geometry (2-Hole)
D.4
HangBag Results
RMSE (L)
Success Rate (t)
Train
Unseen
Unseen
Train
Unseen
Unseen
(OOD)
(OOD)
SD
0.079
0.346
2.041
0,00
0.00
0.00
SP
0.062
0.125
1.886
0.00
0.00
0,00
CD-W
0.053
0.178
3.337
1.00
0.78
0,00
CP-W
0.046
0.173
4.204
1.00
0.80
0.03
CD-NAC
1.254
1.301
1.309
0.56
0.38
0.33
CP-NAC
1.485
1.487
1.486
0.00
0.03
0.00
TAX3D-CD (Ours)
0.183
0.204
0.245
0.94
0.83
0.78
TAX3D-CP (Ours)
0.173
0.192
0.233
0.94
0.93
0.78
DP3
1.00
0.73
0.00
Table 11: HangBag: Fixed Cloth Geometry
D.5
Additional Visualizations
The following pages contain visualizations of our method (as well all bascline methods) across classes of cloth geometry (single- and double-hole) on out-of-distribution scene configurations. For every visualized prediction, both the cloth and configuration were unseen by the model during train- ing.
Within each table row, the top displays the result of the evaluation policy rollout on the correspond- ing model's predicted cloth configuration; the bottom displays the predicted configuration itself. Zooming in may be necessary to properly view the policy executions. For more visualizations, including videos of the policy rollout and the full reverse diffusion process, see our project page.
20

---

D.5.1 Single-Hole Cloth, Unseen Configuration (Out-of-distriibtiion)
SD
CD-W
CD-P
CD-NAC
TAX3D-CD (Ours)
TAX3D-CP (Ours)
21

---

D.5.2 Double-Hole Cloth, Unseen Configuration (Out-of-distribution)
SD
CD-W
CD-P
CD-NAC
TAX3D-CD (Ours)
TAX3D-CP (Ours)
22


Reviews:
[{"review_id": "lZnEohQaff", "reviewer": "Reviewer_QqMb", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper aims to address the problem of robotic manipulation for multimodal, deformable object placement tasks. The authors propose a deep neural network to predict a goal object configuration as dense per-point displacements given initial object and anchor point clouds. Deformation is achieved by per-point (as opposed to per-object) displacement. Multimodality is achieved by a diffusion transformer approach. Claimed contributions are the problem formulation, main method implementation, and a novel benchmark. The paper stops at providing a goal object configuration. A policy for task execution based on this is hinted at but not specified (neither in the paper nor the referenced Appendix C). Training data and experiments are based entirely on simulation. No physical robot (real or simulated) is involved. Experiments are done for a single task: hanging a piece of cloth on a hanger. Success rates are high (95-98%) for a fixed and medium (63-70%) for random cloth geometry.", "summary_of_recommendation": "I consider the lack of modeling of a physical robot (real or simulated) to be a blocker. The hard problems arising from the physics of the robot, the object and the anchor are not even mentioned. It is thus unclear whether the proposed formulation is a contrived problem or actually useful in a real application. The fact that the authors do not relate their approach to related work on deformable object manipulation and the limited experiments do not help with this either.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 1, "robotics_focus": 2, "quality_of_limitations": 1, "questions_for_rebuttal": "Please modify the paper to address the above weaknesses.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aij85PEaeY", "reviewer": "Reviewer_3d53", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This study proposes a new problem called cross-displacement, which extends the existing relative placement to non-rigid objects. This extends transformations to dense transformations to accommodate non-rigid objects. To evaluate this problem setting, a new benchmark based on DENO is proposed, and the effectiveness of the proposed method based on a diffusion model is demonstrated by comparing it with several baselines.", "summary_of_recommendation": "I think the paper can be acceptable as in current format, but can be strengthen significantly by polishing the description about the proposed method and robotics experiments.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "See weakness section", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nxyD3WWzEr", "reviewer": "Reviewer_6Ki8", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper leverages diffusion transformers to perform cross-displacement prediction (an extension of relative placement) between an object point cloud and an anchor point cloud.", "summary_of_recommendation": "Thank you to the authors. The paper is very clear and addresses a problem that is quite under researched. Also the method is cool and uses recent advancements in diffusion models that have shown to perform well in various robotics settings. The main concern which is the reason for the recommendation is the weakness of experiments. Method is evaluated only in simulation on one single task which seems to be contrived (cloth with one or two square holes being hung from a hanger without adhering to gravity). Suggest strengthening experiments in a more realistic setting and on a few different tasks involving deformables in order to demonstrate wider applicability of the method.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Fig. 1: What is the square hole on the cloth? Typically cloths are one piece and could be hung on any spot on the fabric which would make the problem a lot more multimodal. I can’t tell why this example is multimodal if the cloth is only assumed to be hung via the existing hole. \n\n2. How realistic are your assumptions if this method were to be applied in real settings?\n\n3. Why center-mean the goal prediction on the anchor frame and not the action context frame? Latter seems more intuitive if you’re only predicting one Dx set to bring PA to PB. Or are you predicting multiple Dx sets in the style of trajectory waypoints in which case PB would make more sense as the anchor is stationary through the trajectory? (I don’t believe you predict multiple but just to clarify).\n\n4. Is PA hat not the same thing as PA*? I don’t see a difference. If so, please consider using consistent notation. Edit: later on in the paper, PA hat is used for the prediction where PA* is used for the goal (so GT?) therefore in ln. 163 and 165 was the notation supposed to be PA*? \n\n5. Ln. 179-180: Out of curiosity, did you also try just full attention over action features and anchor features as opposed to cross attention? Wondering how the results change. \n\n6. Ln. 186-187: What are the ranges of values of PA_hat vs the range for Dx in your experiments? I’d expect that if PA_hat was significantly larger range/values than Dx, this would make some notable difference in performance. \n\n7. How are you normalizing PA_hat and Dx in your experiments? Normalization in diffusion models tends to have a huge impact. \n\n8. Ln. 187-188: Did you notice any difference between using adaLN blocks vs not? \n\n9. How do you explain that Unseen OOD has a better success rate than Unseen for the 1-2 hole experiment?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summarizing the strengths and weaknesses pointed out by the reviewers:

Strengths:
- The proposed method is an interesting and novel extension of prior work
- Well-written and easy to follow
- Proposed problem is a reasonable and good extension of prior problem settings

Weaknesses:
- All simulation experiments, not real world experiments
- Problem formulation is narrower than it needs to be
- Problem ignores intermediate states between initial and goal
- The way pointclouds are used is problematic, e.g., perfect clouds without occlusions are assumed
- Experiments don't include enough variation to justify generalization. Only evaluated on 1 task in simulation
- No comparisons with other baselines from the literature
- Not enough justification of the chosen architecture
- The description of novelty is missing
- The use of RSME as a metric may not yield the desired results

Overall, 2 reviewers recommend rejection (weak) and 1 acceptance (also weak). The reviewers brought up several valid criticisms that the authors need to address. The authors should carefully read each review and address the concerns as best as possible.

**Updated after rebuttal**: The authors' provided an extensive rebuttal, including changes to the paper and additional experiments. None of the reviewers replied after the rebuttal, so we must analyze the rebuttal to see if it adequately addressed the reviewers' concerns. Fixing the link to the project webpage was a major upgrade to the paper. It provides significant clarity to the methodology and results. From the videos, it is clear that not only are there real robot experiments, but that the method isn't simply predicting a goal pose, the target object is moved along a trajectory to hang it on the peg. Furthermore, the authors added an experiment hanging a bag in both sim and real, which doubles the number of tasks (previously there was only 1).

I find that the authors' changes during the rebuttal has significantly improved the paper and addressed most of the major concerns of the reviewers. Pre-rebuttal the reviewers were 1 weak accept and 2 weak rejects. However, with the new changes I believe this paper should be accepted *WITH ONE MAJOR CAVEAT*: The additional experiments in the second pdf of the rebuttal must be incorporated into the paper. Given the paper is already 3/4 page over the limit, it's likely many details will need to be moved to the appendix, but at least the results should be in the paper proper (never bury good results in an appendix).
Decision: Accept