Rearrangement Planning for General Part Assembly
Authors: Yulong Li
Andy Zeng
Shuran Song
Venue: CoRL 2023 Oral (2023)
Keywords: robotic assembly
pose estimation
3D perception

Abstract: Most successes in autonomous robotic assembly have been restricted to single target or category. We propose to investigate general part assembly, the task of creating novel target assemblies with unseen part shapes. As a fundamental step to a general part assembly system, we tackle the task of determining the precise poses of the parts in the target assembly, which we term “rearrangement planning”. We present General Part Assembly Transformer (GPAT), a transformer-based model architecture that accurately predicts part poses by inferring how each part shape corresponds to the target shape. Our experiments on both 3D CAD models and real-world scans demonstrate GPAT’s generalization abilities to novel and diverse target and part shapes.

Rearrangement Planning for General Part Assembly
Yulong Li!
Shuran Song' 'Columbia University Google Deepmind
Andy Zeng²
tttts::/generrl-part--eae assteaall www.ggithu............... io/
Most successes in autonomous robotic assembly have been restricted to single target or category. We propose to investigate general part assembly, the task of creating novel target assemblies with unseen part shapes. As a fundamental step to a general part assembly system, we tackle the task of determining the precise poses of the parts the target assembly, which we term "rearrangement planning". We present General Part Assembly Transformer (GPAT), transformer- based model architecture that accurately predicts part poses by inferring how each part shape corresponds to the target shape. Our experiments on both 3D CAD models and real-world scans demonstrate GPAT's generalization abilities to novel and diverse target and part shapes.
Abstract:
1 Introduction
Novel Target
The ability assemble new objects hall- mark of visuo-spatial reasoning. With the men- tal image of novel target shape, one can ar- range possibly unseen parts at hand to create a resembling assembly, either building an alien spaceship with lego blocks or a rain shelter with stones. Building autonomous robotic systems that exhibit these capabilities may give rise to wide range of robotics applications from au- tonomously assembling new objects in man- ufacturing plant to building shelter in disaster response scenarios.
Figure 1: General Part Assembly. We seek build autonomous robotic systems can assemble novel target with previously unseen parts. The visualizations are actual imputs and prediction of our model.
Despite the interest and progress in part assembly, existing methods tend to focus on specialized part assembly consisting of fixed targets [1, or seen categories [3, 4J. We propose to instead investigate the task of general part assembly, which takes in inputs both target shape and variable set of part shapes to build an assembly resembling the target. Instead of restricting to fixed objects categories, we require the robotic system to generalize to novel target shapes without additional annotation or supervision. Moreover, the available parts are not guaranteed to be carefully manufactured, and the robotic system has to use parts of slightly differing shapes, the non-exact parts, e.g., building a table with rectangular blocks given a round table as the target.
The task of general part assembly is extension of specialized part assembly that focus fixed targets or categories. For fixed-target assembly, the target shape information is implicitly provided to the agent. A general part assembly agent can also solve category-level part assembly by taking in as input a single instance of the category, while a typical learning method is trained on a large number of instances from the category [3, 4]. For example, given a single table instance, general part assembly agent can assemble tables with either rectangular tabletop or round tabletop.
In this work, we focus the initial perception and planning module for general part assembly, which outputs the precise poses of the parts in the target assembly. Our key insight is to formulate this module as a goal-conditioned shape rearrangement problem, whereby the target can be viewed as a desired 3D shape layout. Consequently, we term this module "rearrangement planing", which
7th Conference Robot Learning (CoRL 2023), Atlanta, USA.

---

aligns with established definitions previously proposed by the Task and Motion Planning (TAMP) community [5]. With this insight, the module factorizes into two steps: predict a segmentation of the target, where each segment corresponds to a part, and infer the pose of each part with pose estimation. To predict accurate segmentation of the target, a key challenge to deal with the ambiguities the target shape due geometrically equivalent parts (e.g.. legs of table). To infer accurate segmentations, we propose General Part Assembly Transformer (GPAT), transformer- based model architecture, that processes input shapes in a fine-to-coarse manner, thereby ensuring consistent segmentation results.
To train and evaluate our model, we build a benchmark based on PartNet [6], a large-scale dataset of 3D objects with part information. We programmatically generate primitive part shapes as non- exact parts. We demonstrate that GPAT generalizes well to entirely new target structures random orientations and novel parts that non-exact matches both synthetic and real-world data.
In summary, our primary contributions are three-fold:
We propose the task of general part assembly to study the ability of building novel targets with unseen parts and create benchmark based on PartNet [6].
We tackle the planning problem for general part assembly as a goal-conditioned shape rearrange- ment problem treating part assembly object segmentation task.
"open-vocabulary (i.e., vocabulary of parts) target
We introduce General Part Assembly Transformer (GPAT) for assembly planning, which can be trained to generalize to novel and diverse target and part shapes.
We believe that GPAT is an exciting step for general part assembly - we discuss both its capabilities and limitations in the report.
2 Related Work
Specialized Part Assembly. A number of learning-based methods have been proposed for part assembly, but they usually have limited generalization abilities, so we refer to them as special- ized part assembly. Reinforcement learning (RL) has success in building part assembly for fixed targets [1, 2, 7, 8) or seen categories [9]. They require costly trial-and-error real-world or physics- based environments to extend to novel targets, and often require low-level state information during training and testing. Another line of work directly works with visual perception and learns shape correspondences, which has success in tasks like kit assembly [10, 11] and shape mating [12]. but they have not tackled part assembly which involves more complex and diverse targets and parts. Previously, part assembly with category-level generalization is tackled with models based on graph neural network (GNN) backbones [3, 4, 13]. Notably, Li et al. [3] and our method shares the same high-level idea of segmenting the target shape, even though their targets are represented as images. Additionally, Funk et al. [14] proposed a full robotic system based on GNN and RL to assemble arbitrary target blueprints with rectangular blocks. In this work, we propose to tackle general part assembly with novel and semantically grounded target and part shapes.
Part Assembly with Object Models. Physics-based part assembly assumes precise models, and the goal poses of the parts are explicitly given or implicitly derived. However, these requirements hinder quick generalization to novel targets and parts. We directly work with visual perception, the 3D point clouds, to predict the precise poses of the parts. With our prediction, one may apply physics-based assembly sequence planning [15, 16, 17, 18] and path planning [19, 20, 21, 22] obtain a complete assembly plan.
Point cloud Registration. Point cloud registration estimates the transformation matrix between two point clouds from different views of the same 3D scene. It is traditionally solved by optimization- based methods and recently by learning-based methods [23]. If we represent target and part shapes as point clouds, general part assembly akin point cloud registration, but it crucially demands optimizing the part poses concurrently. As demonstrated in Sec. 4, basic alterations point cloud registration methods can't directly address general part assembly.

---

Approach
Given target point cloud T and part point clouds (Pt) |=1 as imputs, where N denotes the number of input parts and varies for different shapes, the goal of our task is to predict 6- DoF part pose qi E (3) for each input part P, P, forming a final part assembly, p = U qi (P)), where q: (P;) denotes the transformed part point cloud. To tackle this problem, solve part assembly in two steps: target seg- mentation (Sec. 3.1) which utilizes General Part Assembly Transformer (Sec. 3.2) to de- compose the target into disjoint segments, each representing a transformed part - and pose esti- mation (Sec. 3.3) obtain the final part poses.
segmentationn prodiction
msembll prediction
Pose Estmatiin
MLP
MEP
propose to
CrossAttestion
MultHHeu/AAteerinn
PointNer
PointNes-MaxPool
T
T
3.1 Part Assembly by Target Segmentation
Given target point cloud T and part point clouds (Pil as inputs, we want to segment the target point cloud such that each segment corresponds to part. From the segmentation, we can infer the part pose with pose estimation, which transformation from the part to the corresponding segment.
point cloud
clouds
Figure Method Overview.
Formally, we want to predict set of disjoint segments {77) such that T = and TTT T, = for # Further (T, P represents bipartite matching between the segments and the parts. Note that T. 0 may be empty, suggesting that the part P; is not used in the assembly. The goal of target segmentation is to maximize the geometric resemblance for each pair with non-empty T, as defined by the following minimization problem.
arg min N (T
Σ
min dist (P))
iml Ti.... IN 1=
where dist is some distance metric for point clouds (e.g., chamfer distance). This may appear to be detour since the goal of the task is the transformation di, which is implicitly optimized over. Nevertheless, a model can approximate min, dist (Ti, (P:)) by learning rotationally invariant representations of point clouds to avoid optimization over part poses. Finally, infer part pose by minimizing dist(T, (P))).
3.2 General Part Assembly Transformer (GPAT)
The input General Part Assembly transformer is target point cloud and set of part point clouds, and therefore model backbones designed for single point cloud is insufficient for the task. GPAT uses PointNet [24] to extract initial features for the point clouds, and then leverages transformer-based architecture [25] to jointly optimize over the target shape and the part shapes. To predict accurate target segmentation, a key challenge is the ambiguities in the target shape (e.g., the four legs of a chair are interchangeable). As a result, a target shape often admits multiple ground-trutt segmentations, and fine- mne-grained and consistent segmentation the target required for successful assembly. In light of this, we design the GPAT layer fully exploit the spatial structure of the target point cloud from fine-to-coars, manner, which is inspired the increasing receptive field of convolutional neural networks [26] and progress in hierarchical feature learning for point clouds [27, 28).

---

More formally, let the hidden dimension for features be h. For a query feature q R² and a set of k key features K E R we denote the dot-product attention operator as
(k (g)())
Attention( q. K) = W, softmax (of)tmaa
Vh
where Wg. WR. W, are MLPs.
Given a target T and set of parts /2) GPAT uses Pointnet to extract an initial target point feature v, for each point X, E 'T and an initial part feature uf for each part Pi (with max pooling). Then the features pass through GPAT layers. At the (n + 1)-th GPAT layer, we have a target point feature vi for each point X, T and part feature un for each part P,. The features are updated in three steps. The first step multi-scale attention which parameterized by a positive integer k and denoted by MultiScale in Fig. 2. It updates the target point features follows:
Attention( vy, Nxx(v())
where N&(v,) is the features of the k nearest neighbors of the target point X, GPAT gradually increases k let each point receive global information of the point cloud. The second step the multi-head attention [25], which updates the part features. In the final step, GPAT applies attention updates between the target point features and part point cloud features, denoted CrossAttention in Fig. 2:
ylt Attention v." U*) untt Attention u. V")
where V denotes all target point features and U" denotes all part point cloud features. Finally, GPAT models how likely point X, is matched with an input part P; as
P[X, E 7 Wr(vf) Wr(u
Σn- Wr(x+) Wr(u))
where Wy and Wp are MLP projections.
Data Augmentation. In order generalize targets random poses, we augment the dataset by randomly rotating the target point cloud but keep the order of the points. Thus the ground truth segmentation label is unchanged, which encourages the model obtain rotationally invariant feature representations for the target. We always preprocess the input parts that their principle axes are aligned with world axes. Further, to generalize to unseen categories and non-exact parts, the dataset needs to comprise diverse shapes. We programmatically generate rectangular and spherical primitive shapes of various sizes. For each data sample with exact parts, we construct new sample with each exact part replaced with the primitive of the most similar sizes. GPAT trained with both data samples of exact and non-exact parts. Assemblies with non-exact parts be found in Fig. 3.
Training and Loss. To supervise GPAT, we use the per-point cross entropy loss between the predicted distribution over all parts and the ground truth label. Denote the ground truth segmentation by (T.8* iim and the loss function is
N
L Σ logP|x, E TT
For part assembly, there are often multiple ground-truth labels due geometric equivalence be- tween parts (e.g., legs of a chair). We enumerate all permutations of labels corresponding to the geometrically equivalent parts and adopt the lowest possible cost.
3.3 Predicting Part Assembly with Segmentation
Given a set of parts (P and a segmentation of the target (Ti), i=1" we can find the 6-DoF part pose E SE(3) for each part with pose estimation. Since the parts in our task are not necessarily exact, we use priented-boundingg boxes to estimate part poses which is simple and robust. For each non-empty T, we use principle component analysis to find the oriented bounding boxes of P; and T to solve for qi- In practice, improve the bounding box predictions by filtering the outliers (points that are at least one standard deviation away from the center) in Tí

---

Unseen Calegory
Canonical Powe
Random Pose
Canonical Posc
Random Posc
Precise Part
Imprecise Part
Precise Part
Imprecise Part
Precise Part
Impeecise Part
Prociso Part
Imprecise Part
CD
PA
SR
CD
PA
SR
CD
PA
CD
PA
SR
CD
PA
SR
CD
PA
SR
CD
PA
CD
PA
SR
Opt
7.9
18.3
24
10.0
16.0
0.9
6.3
22.9
21.0
28
5.1
23.1
5.7
54
21.3
4.7
4,1
31.3
28.4
4.2
Go-ICP
72.9
42
0.1
67.9
3.9
0.0
72.3
4.4
0.1
66.2
399
0.0
49.
2.0
0.0
42.6
2.6
0.0
45.7
2.2
0.0
39.3
2.6
0.0
GeaIF
54.3
14.5
4,1
63.4
9.5
1.6
53.7
14.9
4.2
62.6
10.1
1.9
57.3
2.8
0.1
53.8
2.3
0.0
57.4
2.9
0.1
53.9
0.2
0.0
86.3
0.9
0.7
52.0
56.4
DGL
21.5
45.4
10.9
18.0
51.6
14.4
86.7
1.1
0.0
75.5
1.6
0.2
27.2
13.7
1.1
22.1
18.2
0.7
18.9
1.6
0.0
45.0
0.0
DGL
53.4
6.7
0.6
44.3
8.1
0.5
52.3
7.0
0.6
44.0
8.7
0.2
31.3
7.3
0.1
26.4
9.3
0.3
28.4
8.5
0.2
23.9
11.1
0.3
Reg
3.
33
47.9
16.8
11.5
45.4
14.4
57.8
21.5
31.8
5.1
12.3
33.3
4.9
14.1
28.0
Ours
7.6 61.6
23.2
722
64.8
26.0
7.8
60.8
21.7
7.8
64.3
26.0
7.1
53.4
20.1
6.6
56.3
21.7
7.6
52.2
18.8
6.9
55.6
19.8
Table I: Quantitative Results and Comparisons. We adopt three metrics: chamfer distance (CD) measured
Rc, part accuracy (PA) measured in %, and success rate (SR) measured in %..
Evaluation
Tasks. For both training and quantitative evaluation, we use PartNet [6], a large-scale dataset of 3D objects with fine-grained and instance-level 3D part information. We use chairs, lamps, and faucets for training and hold out tables and displays novel categories. We deal with the most fine-grained level of PartNet segmentation, and adopt the default train/test split of the PartNet, which contains 2463 instances of chairs, 1553 instances of lamps, and 510 instances of faucets. We categorize generalization scenarios across three dimensions.
Novel target instances or categories:
We evaluate on the unseen instances of chairs, lamps, and faucets, and two novel categories: tables and displays.
Random target poses: We evaluate on targets at either canonical orientation (as defined in the dataset) or a random orientation uniformly sampled from SO(3).
Non-exact parts: Besides exact parts from the dataset, programatically generated rectangular and spherical blocks non-exact parts. Sample instances can be found in Fig. 3.
Metries. For all the tasks, we measure the quality of the predicted assembly with three metries: chamfer distance, part accuracy, and assembly success rate.
Chamfer distance (CD): Given two point clouds A. B. the chamfer distance between A and B is
CD Σ min ||x yl min ||x - 1 Σ
YEB
F yEB
We use CD(T,P) as a metric, abbreviated as CD, where T is the target point cloud, and p = UN qi (P;) where qi is the predicted pose for the i-th part.'
Part accuracy (PA): Adopted from the previous work [4], part accuracy is defined as,
N CCDI GT (P.).i(P.)) < Tp
Σ N
where 9," is the ground truth pose of the i-th part, Tp 0.01. This metric indicates the percentage of the predicted parts that match the GT part up certain threshold measured in chamfer distance. Due to possible geometric equivalence between parts (e.g., the legs of chair), we enumerate all possible labels of geometrically equivalent parts to obtain different GT poses and take the highest accuracy value.
Assembly Success Rate (SR): A predicted assembly is considered successful if its part accuracy (PA) is equal to We report the percentage of successful predictions out of all data samples as the assembly success rate (SR).
Algorithm comparisons. Since general part assembly is novel task, there are no previous methods specifically solving the task. We adapt methods for point cloud registration and specialized part assembly and compare with variants of our method for ablation studies.
Note that in the previous work [4], 'shape chamfer distance" is defined differently with T = UN of (P). In tasks, the target is not union of the given parts, so the values according our metric usually larger.
5

---

Opt: Covariant matrix adaptation evolution strategy (CMA-ES) [29] is used to optimize the poses of each part by minimizing the chamfer distance CD as defined above.
Go-ICP: We greedily match each part point cloud to the target point cloud using Go-ICP [30].
GeoTF: Geometric Transformer (GeoTF) [31] is one of the SoTA methods for point cloud registration. We modify the algorithm to simultaneously optimize for all part poses.
NSM: Neural Shape Mating (NSM) [12] uses transformer- based model to solve pairwise 3D geometric shape mating such as reconstruct two broken pieces of an object. We modify their algorithm to match each part to the target and simultaneously optimize for all part poses.
DGL: Dynamic Graph Learning (DGL) [4] tackles category-level part assembly by leveraging iterative graph neural network backbone to regress part poses. Designed for category-level generalization, DGL does not take in the target shape as input. To adapt to our task, include the target encoding as a node into the graph neural network framework. DGL is trained only with targets at canonical poses following the previous work. DGL-aug uses the same training dataset as our model, with augmentation of targets at random poses.
Reg: Instead of predicting segmentation of the target for the subsequent pose estimation, we replace the final dot-product segmentation layer of our model with MLPs to directly regress 6DoF pose for each part. We trained the modified model with the supervision of GT poses.
. TF: As an ablation, we replace each GPAT layer with vanilla transformer layer [25].
Experimental Results
Tab. and Fig. 3 summarizes the main quantitative and qualitative results, and the following sections provide detailed discussions. Please refer to the supplementary materials for more results.
Part assembly by target segmentation is more generalizable. The optimization baseline (Opt) achieves the lowest chamfer distance (CD) in some scenarios, but its part accuracy (PA) and success rate (SR) are significantly lower. Directly optimizing the part poses often result in predictions at local minima where the predicted assembly matches the contour of the target, but the assembly makes no semantic sense. (Please refer to supplementary materials for an example.)
As classical optimization bassed algorithm, Go-ICP tries to match individual part to the entire target method, GeoTF achieves better performances
which fails with no surprises. As a learning- for seen categories, but fails nonetheless for unseen categories. Since the parts non-exact, it is challenging for point cloud registration methods to find suitable correspondences either in spacial coordinates or learned feature space.
GPAT also outperforms regression-bascd models (DGL, DGL-aug, NSM, Reg) across all the tasks, especially at scenarios that requires more generalization (Tab. and Fig. 3). For the most challeng- ing scenario, regression-Nes
models achieve less than 1% success rate, while GPAT has 19.8% success rate which attained for all the test scenarios. To directly regress poses, a model needs to learn rotationally equivariant features for the target shape. However, given non-exact parts, unseen models fail to capture
categories, and targets at random poses, show that the regression-basse the distribution of poses. They either overfit certain canonical poses and assembly structures or fail model, is trained to learn rotationally invariant
to learn. In contrast, GPAT, a segmentation-base representations of the shapes, and thus it experiences minor performance drops facing generaliza- tion scenarios. Further, training with diverse shapes makes the representations generalizable and applicable to new categories.
GPAT is robust against targets with ambiguities. Compared to the alternative ciamenntationn-asedd model using vanilla transformer, GPAT achieves better results for all test scenarios, especially for unseen target categories (Tab. 1 and Fig. 3). We compare the segmentation accuracy and show the results in Tab. 2 and visualize a typical failure case of TF in Fig. 5. When facing inputs with multi-model ground truths (e.g., a chair with identical legs), TF unable to produce consistent segmentation, which hinders successful assembly prediction. With GPAT layers, we fully leverage the point cloud strueture and process the point cloud in a fine-to-coarse manner, thereby achieving local consistency of the segmentation predictions.

---

Unseen Instance
Unseen Calegory
Canonical
Random Pose
Canonical Pose
Random
Exact
Non-Exact
Exact
Non-Exact
Exact
Non-Exact
Exact
Noa-Exact
Target
GT
Go-ICP
GooTF
NSM
DGL
DGL-aag
Reg
TF
Ours
Figure Assembly Results Comparisons. targets at random poses, targets predictions are transformed to be visualized at canonical poses for better understanding. Please see Fig. 4 for more results of randomly oriented targets. The optimization based approach (Opt and Go-ICP) tends to stuck at local minima. The learning-hased alternatives (GeoTF, NSM, DGL., Reg) overfit the training scenarios and fail to learn rotationally equivariant features for target shapes that necessary for accurate pose inference. The alternative model uses the vanilla transformer (TF) fails produce consistent segmentations for targets with geometrically equivalent parts (see Fig. for a detailed example). With the multi-scale attention layer, GPAT fully leverages the spatial structure of the target point clouds produce consistent segmentations and accurate assemblies.
segmentation-be

---

Figure 4: Assembly Results for Targets at Random Orientations. We show more results for the same targets in Fig. but at random orientations. GPAT is robust against the orientation of the target shape. More results can be found in the supplementary materials.
GPAT solves generalizes well to real-world data. We use the real-world scans from redwood dataset [32] as targets and part point clouds from PartNet. As seen in Fig 6, our method produces diverse assemblies that resemble the target. This result also illustrates how GPAT can be used to assemble different sets of parts given single target shape from the category.
Ours
Real-wordd Scan
Assembly Prodiction
TITO Gnnreeante Aaazzey
Target
Pore
Figure 5: GPAT is robust against ambiguitics.
Unseen Instance
Unseen Category
Canonical
Random Pose
Canonical Pose
Random Posc
Pre
Imp
Imp
Pre
Imp
Phe
Imp
TF
70.0
62.2
76.5
69.2
63.7
62.4
62.9
60.9
Ours
76.7
70.9
76.6
71.1
69.5
69.3
69.6
69.5
Table 2: Segmentation Accuracy (%)
Figure Results on Real-world Data
Failure mode analysis. GPAT is not without limitations, and Fig. 7 shows some typical failure cases. First, GPAT tends to give incorrect segmentation predictions if some parts are hidden inside larger part (c.g., g. the light bulbs in a lamp) or the parts are less separable (c.g.. overlapping parts of microwave). To solve with these issues, it is possible introduce additional information like colors and normals of the point clouds inputs. Additionally, oriented bounding box can be insufficient as pose estimation method for some parts. To tackle this problem, dearning-basee pose estimation module can potentially replace the bounding box procedure.
6 Conclusion
Segmentation Prediction
Assembly Prediction
Target
GT
In this work, we formulate the task of gen- eral part assembly, which focuses on building novel target assemblies with diverse and un- seen parts. To plan for general part assembly task, we propose General Part Assembly Trans- former (GPAT) and factorizes the task into target segmentation and pose estimation. Our exper- iments show that GPAT performs well under all the generalization scenarios. By integrating with an assembly sequence and path planning algorithm, we believe that GPAT has great po- tential building vision-based general robotic assembly systems.
Figure Typical Failure Cases.

---

7 Acknowledgement
We would like to thank Huy Ha, Zhenjia Xu, Cheng Chi, and Zeyi Liu for their helpful feedback and fruitful discussions. This work was supported in part by NSF Award #2037101, #2143601, and #2132519. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
References
[1] G. Thoma,, M. Chien, A. Tamar, J. A. Ojea, and Abbeel. Learning robotic assembly from cad. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3524-3531. IEEE, 2018. 1,2
[2] Y. Narang, Storey, I. Akinola, M. Macklin, Reist, L. Wawrzyniak, Y. Guo, A. Mora- vanszky, G. State, M. Lu, et al. Factory: Fast contact for robotic assembly. arXiv preprint arXiv:2205.03533 2022. 1,2
[3] Y. Li, K. Mo, L. Shao, M. Sung, and L. Guibas. Learning 3d part assembly from a single image. European Conference on Computer Vision, pages 664-682 Springer, 2020. 1, 2, 12
[4] G. Zhan, Q. Fan, K. Mo, L. Shao, B. Chen, L. J. Guibas, H. Dong, al. Generative 3d part assembly via dynamic graph learning. Advances in Neural Information Processing Systems, 33:6315-6322 2020. 1, 2, 5, 6, 12
[5] D. Batra, A. X. Chang, S. Chernova, A. J. Davison, J. Deng, V. Koltun, S. Levine, J. Malik, I. Mordatch, R. Mottaghi, et al. Rearrangement: A challenge for embodied ai. arXiv preprint arXiv:2011.01975, 2020.
[6] K. Mo, Zhu, A. X. Chang, Yi, Tripathi, L. J. Guibas, and Su. Partnet: large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pages 909-9188 2019. 2, 12
[7] S. K. S. Ghasemipour, S. Kataoka, B. David, D. Freeman, S. S. Gu, and I. Mordatch, Blocks assemble! learning to assemble with large-scale structured reinforcement learning. In Interna- tional Conference on Machine Learning, pages 7435-7469, PMLR, 2022.
O. Aslan, B. Bolat, B. Bal, T. Tumer, E. Sahin, and S. Kalkan. Assemblerl: Learning to assemble furniture from their point clouds. In 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2748-2753. IEEE, 2022. 2
[9] M. Yu, L. Shao, Z. Chen, T. Wu, Q Fan, K. Mo, and H. Dong. Roboassembly: Learning gener- alizable furniture assembly policy in a novel multi-robot contact-rich simulation environment. arXiv preprint arXiv:2 10143, 2021. 2
[10] K. Zakka, Zeng. J. Lee, and S. Song. Form2fit: Learning shape priors for generalizable as- sembly from disassembly. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 9404-9410. IEEE, 2020. 2
[11] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin,
D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning, pages 726-747. PMLR, 2021.
[12] Y.C. Chen, H. Li, D. Turpin, A. Jacobson, and Garg. Neural shape mating: Self-supervised object assembly with adversarial shape priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12724-12733, 2022. 2,

---

[13] A. Narayan, R. Nagar, and S. Raman. Rgl-net: A recurrent graph learning framework for the IEEEJCVE Winter Conference on Applications of Computer Vision, pages 78-87, 2022. 2
progressive part assembly. In Proceedings the
[14) N. Funk, G. Chalvatzaki, B. Belousov, and J. Peters. Learn2assemble with structured repre- sentations and search for robotic architectural construction. In Conference on Robot Learning, pages 1401-1411. PMLR, 2022. 2
[15] L. H. De Mello and A. C. Sanderson. A correct and complete algorithm for the generation of mechanical assembly sequences. In 1989 IEEE International Conference on Robotics and Automation, pages 56-57. IEEE Computer Society, 1989.
[16] C. Sinanoglu and H. R. Börklü. An assembly loquence-planning system for mechanical parts using neural network. Assembly Automation, 2005. 2
[17] Y. Huang, Garrett, Ting, S. Parascho, and C. Mueller, Robotic additive construction of bar structures: Unified sequence and motion planning. Construction Robotics, 5(2):115-130) 2021.
[18] Y. Tian, Xu, Y. Li, Luo, S. Sueda, Li, K. D. Willis, and W. Matusik. Assemble them all: Physics-basee bassed planning for generalizable assembly by disassembly. ACM Transactions on Graphics (TOG), 41(6):1-11 2022. 2
[19] R. H. Wilson and J.-C. Latombe. Geometric reasoning about mechanical assembly. Artificial Intelligence, 71(2):371 -396,, 1994. 2
[20] D. Halperin, J.-C. Latombe, and R. H. Wilson. A general framework for assembly planning: The motion space approach. In Proceedings of the fourteenth anmual symposium on Computational geometry, pages 9-18, 1998.
[21] S. Sundaram, I. Remmler, and N. M. Amato. Disassembly sequencing using a motion plan- ning approach. In Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No. 01CH37164), volume 2, pages 1475-1480. IEEE, 2001.
[22] X. Zhang, R. Belfer, P. G. Kry, and F. Vouga. C-space tunnel discovery for puzzle path planning.
ACM Transactions on Graphics (TOG), 39(4):00411, 2020. 2
[23] X. Huang, G. Mei, J. Jhhang and R. Abhas. comprehensive survey on point cloud registration.
arXiv preprint arXiv:2 2021. 2
[24] C. R. Qi, H. Su, K. Mo, and L. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[25] A. Vaswani, Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017. 3, 4.
[26] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84 90, 2017.
[27] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
[28] H. Zhao, L. Jiang. J. Jia, P. H. Torr, and V. Koltun. Point transformer. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages (6259-16268, 2021.
[29] N. Hansen, Akimoto, and Baudis. CMA-ES/pyema Github. Zenodo,
D01:100.881/eemmo.25556666 Feb. 2019. URL hittps:/ddos http:/bbb..or10.5221211///// 2559634. 6
10

---

[30] J. Yang. H. Li, D. Campbell, and Y. Jia. Go-iep: A globally optimal solution to 3d icp point-set registration. IEEE transactions on pattern analysis and machine intelligence, 38(1 1): 2241-2254, 2015. 6, 12
[31] Z. Qin, H. Yu, C. Wang, Y. Guo, Y. Peng, and K. Xu. Geometric transformer for fast and robust point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 11143-11155 2022.
[32] S. Choi, Q.-y. Zhou, S. Miller, and V. Koltun. A large dataset of object scans, arXiv preprint 2016. 8, 17
[33] Z. Wu, S. Song, A. Khosla, Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015. 12
[34] B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar. Benchmarking manipulation research: The ycb object and model set and benchmarking protocols. arXiv preprint 10X.i:::00000 03143, 2015. 12
[35] Y. Wang and J. M. Solomon. Deep closest point: Learning representations for point cloud registration. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3523-3532, 2019. 12
[36] Luo, K. Mo, Z. Huang, J. Xu, S. Hu, L. Wang, and H. Su. Learning to group: A bottom-up framework for 3d part discovery in unseen categories. arXiv preprint arXiv:2002.06478, 2020.
13
[37] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph enn for learning point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019. 15
[38] Yu, K. Liu, Zhang, Zhu, and K. Xu. Partnet: recursive part decomposition network for fine-grained and hicrarchical shape segmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9491-95000 2019. 15
[39] D. P. Kingma and Ba. Adam: method for stochastic optimization. arXiv preprint
arXiv: 141266800,, 2014. 15

---

8 Appendix
8.1 Additional Results and Analysis
Additional Visualization. Fig. 13 and Fig. 14 show additional results on simulated and real-world data, respectively.
Quantitative Results for Categories. Table 3 shows detailed quantitative evaluation for unseen instances for seen categories (Chair, Lamp, Faucet) and unseen categories (Table, Display).
Table 3: Quantitative results of our algorithm different categories.
Canonical Pose
Random Pose
Precise Part
Imprecise Part
Precise Part
Imprecise Part
CD
PA
SR
CD
PA
SR
CD
PA
SR
CD
PA
SR
Chair
7.7 57.7
19.3
7.3
64.4
25.1
7.6
58.4
19.3
8.3
63.0
24.0
Lamp
7.6 66.9
29.1
5.9
72.1
40.9
8.1
64.2
26.2
6.0
74.4
45.5
Faucet
7.3 65.6
24.4
6.5
63.1
20.7
7.4
63.6
20.6
6.5
62.3
22.4
Table
7.5 52.0
20.6
7.0
55.1
21.5
8.1
50.8
17.8
6.9
55.7
22.2
Display 4.2 59.2
23.2
4.7
59.3
20.2
4.4
60.8
23.8
4.9
58.5
16.9
GPAT builds creative assemblies. To fully test the generalization abilities of GPAT, we provide unseen part shapes like a banana, hammers, and forks as parts to create novel targets such as a plane. Our model predicts creative assemblies given target shapes from unseen categories and non- exact parts, seen in Fig. 8, The shapes are taken from PartNet [6], ModelNet40 [33], and YCB dataset [34].
Target
Parts
Segmentation
Assembly
TT
Figure 8: Creative Assemblies. Our model predicts creative assemblies given target shapes from unseen categories and non-exact parts. 1st row: a chair assembled with lamps as chair legs. 2nd row: a table assembled with a plate and spoons. 3rd row: plane assembled with banana and hammers.
Oriented bounding boxes offer sufficient pose estimator. Once we obtain the target segments with GPAT, we compare with alternative methods to obtain the final poses and present the quantitative results in Tab. 4. The alternative methods include Go-ICP [30], DCP [35] (we take the released model pretrained on Modelnet40 [33]). Additionally, we adapt the previous work [3] by Li et al. to our task. Li ei al. consider targets represented as images, and train model to produce 2D part segments and another GNN-based model to regress part poses. We produce 3D segments using our pretrained GPAT and train the GNN-based backbone proposed DGL [4]. an improved model compared to that used in [3]. We find the heuristies based on oriented bounding boxes to produce comparative or better results compared more sophisticated alternative methods. Furthermore, provide further analysis in the Appendix to show that the main bottleneck of the problem is segmentation opposed to pose estimation.
12

---

Unsoon Instance
Unseen Category
Canonical Posc
Random Pose
Canonical Posc
Random Pose
Precise Part
Imprecise Part
Precise Part
Imprecise Part
Precise Part
Imprecise Part
Precise Part
Imprecise Part
CD
SR
CD
PA
SR
CD
SR
CD
PA
SR
CD
SR
CD
PA
SR
CD
SR
CD
PA
SR
GPAT-GoICP GPAT-DCP
6.7
63,9
23.7
63.5
20.0
6.8
64.9
24.6
62.5
19.7
5.9
53,9
19.7
55.7
18.6
5.3
55.8
20
59 58.6 22.1
18.2
57.3
6.2
18.6
52.1
211
18.7
36.0
5.5
184
$2.0
288
15A
35.1
34
16.7
90.1
10
15.3
35.4
4.4
16.9 50.5 2,3
GPAT-DGL
12.1
52.1
13.0
10.8
58.4
12.2
51.
11.5
10.5
55.8
13.1
38.4
6.9
10.5
44.8
9.6
11.5
42.3
10.3
13.6 188] 55.6
10.4
48.5
GPAT-BR (Ours) 7.6
23.2
6488
26,0
7.8
21.7
64.3
26,0
7.1
20.1
6.6
56.3
21.7
7.6
Table 4: Evaluation of the Pose Estimation Module. We find the efficient heuristics based on oriented bounding boxes produce comparative or hetter results compared to more sophisticated alternative methods.
Segmentation Accuracy is the Main Bottleneck. In Fig. 9, we plotted the mean success rate / part accuracy conditioned on the minimum segmentation accuracy. We find that when segmentation accuracy approaches perfect, average success rate and part accuracy approaches 90%, while the current overall numbers are around 20% and 60%, respectively. This shows that segmentation accuracy is still the main bottleneck of our method.
1.0
part accuracy
0.4
0.9
02
Minimum Segmentation Accuracy
Figure 9: Segmentation Accuracy is the Main Bottleneck. Each point the plot reads as "for all the data samples with minimum accuracy of x, the average success rate part accuracy y".
Optimization is prone to local minima. The optimization baseline (Opt) achieves the lowest chamfer distance (CD) in some scenarios, but its part accuracy (PA) and success rate (SR) significantly lower. As seen in Fig. 10, directly optimizing the part poses to match the target often result in predictions at local minima where the predicted assembly matches the contour of the target, but the assembly makes no semantic sense.
GPAT is applicable part discovery. GPAT is directly applicable the task of part discovery, ie., predict part segmentation given target [36], if do not provide input parts. We show some qualitative results in Fig 11 to test GPAT's part discovery abilities. Given non-exact parts, PAT predicts accurate segmentations as usual. If we input identical blocks, which specifies the number of parts but provides little information about the part shapes, then GPAT predicts reasonable segmentations with the specified number of segments. Finally, we omit the input parts, and GPAT successfully discovers parts in the target shape.
GPAT is aware of part scales. Part assembly often involves parts that have the same geometry but different scales, so it is necessary for a model to discriminate parts of different scales to create correct assemblies. As an qualitative illustration in Fig. 12, we adjust the scale of the parts that have same geometry (the legs of chair/table), and the model correctly associates parts of different scales to the target to build the desired assemblies.
13

---

Target
GT
Opt
Ours
Figure 10: Optimization is prone to local minima.
Non Son-Exact
Identical
Input
Non-Exact
Ideeticall
Input
Parts
M
M
Seginemation Predictions
IT
Figure 11: Application to Part Discovery. Given non-exact matching parts (Non-Fxact), identical blocks (Identical), and no part point clouds input, GPAT predicts reasonable part segmentations of the target.
Segmentation Prediction
Assembly Prediction
Target
Parts
uTi
Figure 12: Sensitivity Scale. The legs of the chair/talle are manually scaled, and the model correctly associate parts of the same shape but different sizes.
14

---

8.2 Data and Training Details
We use Furthest Point Sampling (FPS) to sample 1,000 points for each part point cloud and 5,000 points for each target point cloud. Following the previous work [37], we also zero-center all the point clouds, and align the principle axes of the part point clouds with the world axes using Principle Component Analysis (PCA). Additionally, 3-dimensional sizes of the part, and two parts considered geometrically equivalent if they have the same part type as labeled by the PartNet dataset [38] and same sizes up to small threshold.
similarly axis-aligned bounding boxes to obtain
In our training, we down-sample the target point features by a factor 10, so for each sample, we obtain 500 target point features. We use feature dimension of 256 and we use GPAT layers, with values of 16, 16, 32, 32, 64, 64, 500, 500. We use Adam [39] with learning rate of 0.00004, batch size of 36. We train for 2000 epochs in total.
15

---

Canonical Pose
Random Pose
Exact
Non-Exact
Exact
Non-Exact
Chair
Lamp
F T
1/1
alr
Faucet
F ale
MIADL
Table
MT
Display
Figure 13: Qualitative Results. Chairs, lamps, and faucets are seen during the training. Tables and displays are unseen categorics. The first row of each category displays targets in black, and the second row shows our predictions.
16

---

Real-world Scans
Predictions
ILLL न
TIT
Figure 14: Results on Real-world Data. Non-exact parts from same category as the target point cloud, which are teal-world scans taken from Redwood dataset [32].
17


Reviews:
[{"review_id": "z0N4O1ftad", "reviewer": "Reviewer_sArU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This manuscript introduces a transformed-based neural network model for the general part assembly task. The model takes as inputs a complete target point cloud and the set of parts to be assembled,  and outputs the segmentation of the target point cloud and associated parts for each segmented region. Then a pose estimation module is utilized to generate the part pose. The authors also conduct extensive experiments to verify the effectiveness of the proposed pipeline and compare it with other methods.", "summary_of_recommendation": "This paper proposes a transformed-based neural network to tackle the general part assembly task by formulating it as a goal-conditioned shape-matching problem, which is kind of novel. However, the topic is more related to computer vision instead of robotics.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Q1. See weakness 2.\n\nQ2. Section 4. Line 184. “Due to possible geometric equivalence between parts (e.g., the legs of a chair), we enumerate all possible labels of geometrically equivalent parts to obtain different GT poses and take the highest accuracy value.” Does the dataset of Part provide geometrically equivalent parts annotation? If not, how do the authors gather such annotations?\n\nQ3. In Section 5, Line 242. “We use the real-world scans from redwood  dataset [29] as targets and part point clouds from PartNet.” What is the performance if both the target point cloud and part point clouds are scanned from the real world?\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. The paper is well-written and easy to follow.\n2. The paper introduces a transformed-based framework to tackle the part assembly task with convincing results both qualitatively and quantitively. \n\nWeakness:\n1. The title of the paper “general part assembly PLANNING” is misleading. There is no classical planning incorporated in this manuscript.\n2.  It’s not clear to the reviewer how effective the proposed pose estimation using the oriented-bounding boxes. It would strengthen the paper if more analysis and comparison is provided.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "qT5oXonofb", "reviewer": "Reviewer_JQms", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper formulates the general part assembly problem as a goal-conditioned shape rearrangement task, and proposes a Transformer-based architecture that processes input point clouds in a fine-to-coarse manner in two steps: predict a segmentation of the target shape and subsequently infer the pose of each part. Experiments on PartNet demonstrate the method and show its superior performance compared to baselines. Additionally, generalization to real-world scans is shown on the redwood dataset.", "summary_of_recommendation": "A strong high-quality paper that proposes a general approach for solving part assembly problems on point clouds. In spite of the demonstrations being presented on a restricted set of categories, the results appear convincing and promising. Combined with task and motion planning, this method can contribute to a general autonomous robotic assembly system.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The paper argues that RL \"has success in building part assembly for **fixed targets** [...] or **seen categories** [...]\" (lines 64--65), whereas this paper allows for generalization to \"**novel target shapes** without additional annotation\". However, I would like to point out that there was also a demonstration of RL+GNN method which can assemble various shapes even with a robot arm. Namely, citing from the abstract of [1]: \n\n\"Combining structured representations with model-free RL and Monte-Carlo planning **allows agents to operate with various target shapes** and building block types. We design a hierarchical control framework that learns to sequence the building blocks to **construct arbitrary 3D designs** and ensures their feasibility, as we plan the geometric execution with the robot-in-the-loop.\"\n\nOf course, the method proposed in the current submission is quite different, as it works on point clouds and on more semantically grounded shapes which have parts, whereas the referenced paper builds arbitrary shapes out of given blocks. Nevertheless, both approaches tackle the part assembly problem, and I would recommend to slightly rephrase the introduction and related work section on Specialized Part Assembly to acknowledge that other methods also attempted to generalize to novel shapes, and emphasize how the proposed method is different.\n\n[1] Funk, N., Chalvatzaki, G., Belousov, B., & Peters, J. (2022). Learn2assemble with structured representations and search for robotic architectural construction. In Conference on Robot Learning (pp. 1401-1411). PMLR.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "High quality paper, clearly written, original, and providing a significant step forward on the part assembly problem.\n\nThe evaluation is a bit limited, as the method was trained on \"chairs, lamps, and faucets\" and tested on \"tables and displays\". Is there a reason why only so few categories were used? What would happen if one swaps some categories between train and test sets (e.g., swap tables and chairs)?", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "psQYRlgkZO1", "reviewer": "Reviewer_wPeg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a novel method for tackling the task of 3D shape part assembly planning that can generalize across category boundaries. Given as inputs a 3D target shape point cloud and a set of 3D part shape point clouds, the system outputs the part segmentation and then the part poses of the input 3D parts to assemble the target shape. The parts can be the exact parts or non-exact parts (e.g., using 4 spoons and a cuboid to assemble a table shape). The core idea is to use PointNet to extract per-point features in the input shape and per-part features among the input parts and then employ a multi-attention transformer across the shape points and parts first to perform part segmentation and then optimize for the output part poses. The authors compared the proposed method to several baseline methods using the PartNet parts and show better and more generalizable results than baselines. Results on real scans are also presented. Failure cases and limitations are well discussed.", "summary_of_recommendation": "Overall, this is a good paper proposing a new interesting and important problem and presenting a technically novel method with good experimental results supporting its contributions. I'd like to see my questions addressed in the rebuttal, but they are minor points only serving as suggestions on potentially improving the paper.\n\nPost rebuttal: Thanks for authors' rebuttal. I keep my score as Weak Accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Besides the above points in weaknesses, I have two clarification questions:\n- in Fig. 6, the second row, why the back slats are not fitted?\n- in Fig. 2, if the two \"wing\" parts are the same geometry, how to learn different features for them so that the green one corresponds to the left wing, and the blue one corresponds to the right wing, instead of the other way around?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The paper presents a novel and interesting task of general part assembly.\n- The proposed method is technically novel, specifically in proposing to do segmentation first and using optimization-based pose estimations for better generalization capabilities. \n- Experiments show clear advantages over baseline methods both qualitatively and quantitatively.\n\nWeaknesses:\n- It's a bit unclear why the proposed method would not overfit the training categories as the baselines do, since the PointNet/transformer backbones may also be subject to overfitting as the baseline methods. Or in other words, may the baseline methods also be able to generalize well to novel categories with increased numbers of the training categories? Can the authors show results on more test categories that are dissimilar to the training categories?\n- The proposed method is similar to [3] which also proposed to do segmentation first but it's in 2D. It would be better to add a comparison to an adapted version of [3] to segment the input shapes in 3D.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "KAi_EtYlm3", "reviewer": "Reviewer_v4vx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a new task consisting of building a previously unseen target objects using a list of new parts specified as point clouds.  In the authors' proposed problem, the parts may not exactly fit the target model, so the network must learn to assemble something as close as possible to the target shape.  The authors also propose a transformer-based model to address this task using multi-scale attention and cross-attention to part representations.", "summary_of_recommendation": "The paper is well written and executed, but the limited originality and lack of robotics focus weigh it down.  I do not feel strongly about rejecting this paper though, and would be willing to adjust my score upwards if the authors or more importantly the other reviewers make a strong case that these limitations are not important to the CoRL audience.\n\nPost Rebuttal: I am bumping this up to weak-accept.  I appear to have the lowest review of this paper, and my complaints were about the originality and the lack of robotics focus.  I wasn't sure if this was the right way to evaluate the paper, and these issues didn't seem to have been a problem for the other reviewers, so I am adjusting upward.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. How could the system be adapted to problems requiring sequential planning?\n2. How would this be incorporated into a larger robotic system?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n+ The paper is clearly written and easy to understand\n+ The problem is interesting and important\n+ The authors present many ablations, although these could be improved (see below)\n+ The proposed solution appears well executed\n\nCons:\n- (MINOR) The authors call this an assembly problem, but in reality it is closer to object decomposition.  The model only predicts a single point-cloud to part assignment, without any step-by-step analysis for how to construct the object, assuming that the parts may be connected in any arbitrary order by simply moving them to the correct 3D pose.  Similarly Figure1 labels the model's estimate an \"Assembly Plan.\" The word \"plan\" suggests a sequential set of actions, which is not considered here.  While similar problems in prior work have also labelled themselves \"assembly\" problems, it would be beneficial to make this distinction clear by instead using terms like \"decomposition\" and \"estimation\" rather than \"assembly\" and \"plan.\"\n- (MODERATE) The authors' method consists of two steps: first compute an assignment from point cloud to the provided parts, then use a very simple bounding-box alignment mechanism to estimate the pose of each part.  With this in mind, the ablations would be better organized if these two components were isolated and handled independently.\n- (IMPORTANT) Limited originality.  The high level problem and solution concept is quite similar to Li 2020 (citation [3]), but uses point-clouds instead of an image, with previously unseen categories.  The model components are also fairly standard which does not make a strong case for the practical utility of the paper.\n- (IMPORTANT) While this might be used as a component of some robotics system, the connection to robotics in this paper is somewhat limited.  No experiments with real or simulated robots are presented.  However, I am not sure that robotics experiments would improve the quality of the paper, since the focus seems closer to 3D vision.  A vision conference may be a better venue for this work.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel approach to the problem of 3D shape part assembly planning by formulating it as a goal-conditioned shape rearrangement task and proposing a Transformer-based architecture. The proposed method shows promise in solving this task, and it has been received positively by reviewers, with both strengths and areas for improvement highlighted.
Strengths identified by reviewers include the quality of the paper, its clarity, and the originality of the approach. The method's technical innovation and potential for impact in the field have been highlighted. The paper also addresses limitations clearly and provides solid experimental results.
Reviewers have raised questions and suggestions regarding the evaluation setup. Specifically, concerns were raised about the limited number of categories used for testing and the potential effects of swapping training and testing categories. The authors have responded by explaining the limitations of available dataset categories and conducting a preliminary experiment to evaluate the impact of category choices.
Another point of discussion is the similarity of the proposed method to prior work in terms of generalizing to novel shapes. A related work by Funk et al. [1] was brought to the authors' attention, demonstrating the importance of acknowledging such related works and highlighting the differences between them.
The paper is considered to be of high quality, presenting a significant step forward in addressing the part assembly problem with promising results and innovation.
Post-rebuttal comments show that reviewers have maintained their recommendations, indicating that the authors' responses effectively addressed their concerns and reinforced the paper's value.
Decision: Accept (Oral)