Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This manuscript introduces a transformed-based neural network model for the general part assembly task. The model takes as inputs a complete target point cloud and the set of parts to be assembled,  and outputs the segmentation of the target point cloud and associated parts for each segmented region. Then a pose estimation module is utilized to generate the part pose. The authors also conduct extensive experiments to verify the effectiveness of the proposed pipeline and compare it with other methods.
review: Strengths:
1. The paper is well-written and easy to follow.
2. The paper introduces a transformed-based framework to tackle the part assembly task with convincing results both qualitatively and quantitively. 

Weakness:
1. The title of the paper “general part assembly PLANNING” is misleading. There is no classical planning incorporated in this manuscript.
2.  It’s not clear to the reviewer how effective the proposed pose estimation using the oriented-bounding boxes. It would strengthen the paper if more analysis and comparison is provided.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper proposes a transformed-based neural network to tackle the general part assembly task by formulating it as a goal-conditioned shape-matching problem, which is kind of novel. However, the topic is more related to computer vision instead of robotics.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors propose a new task consisting of building a previously unseen target objects using a list of new parts specified as point clouds.  In the authors' proposed problem, the parts may not exactly fit the target model, so the network must learn to assemble something as close as possible to the target shape.  The authors also propose a transformer-based model to address this task using multi-scale attention and cross-attention to part representations.
review: Pros:
+ The paper is clearly written and easy to understand
+ The problem is interesting and important
+ The authors present many ablations, although these could be improved (see below)
+ The proposed solution appears well executed

Cons:
- (MINOR) The authors call this an assembly problem, but in reality it is closer to object decomposition.  The model only predicts a single point-cloud to part assignment, without any step-by-step analysis for how to construct the object, assuming that the parts may be connected in any arbitrary order by simply moving them to the correct 3D pose.  Similarly Figure1 labels the model's estimate an "Assembly Plan." The word "plan" suggests a sequential set of actions, which is not considered here.  While similar problems in prior work have also labelled themselves "assembly" problems, it would be beneficial to make this distinction clear by instead using terms like "decomposition" and "estimation" rather than "assembly" and "plan."
- (MODERATE) The authors' method consists of two steps: first compute an assignment from point cloud to the provided parts, then use a very simple bounding-box alignment mechanism to estimate the pose of each part.  With this in mind, the ablations would be better organized if these two components were isolated and handled independently.
- (IMPORTANT) Limited originality.  The high level problem and solution concept is quite similar to Li 2020 (citation [3]), but uses point-clouds instead of an image, with previously unseen categories.  The model components are also fairly standard which does not make a strong case for the practical utility of the paper.
- (IMPORTANT) While this might be used as a component of some robotics system, the connection to robotics in this paper is somewhat limited.  No experiments with real or simulated robots are presented.  However, I am not sure that robotics experiments would improve the quality of the paper, since the focus seems closer to 3D vision.  A vision conference may be a better venue for this work.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is well written and executed, but the limited originality and lack of robotics focus weigh it down.  I do not feel strongly about rejecting this paper though, and would be willing to adjust my score upwards if the authors or more importantly the other reviewers make a strong case that these limitations are not important to the CoRL audience.

Post Rebuttal: I am bumping this up to weak-accept.  I appear to have the lowest review of this paper, and my complaints were about the originality and the lack of robotics focus.  I wasn't sure if this was the right way to evaluate the paper, and these issues didn't seem to have been a problem for the other reviewers, so I am adjusting upward.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents a novel method for tackling the task of 3D shape part assembly planning that can generalize across category boundaries. Given as inputs a 3D target shape point cloud and a set of 3D part shape point clouds, the system outputs the part segmentation and then the part poses of the input 3D parts to assemble the target shape. The parts can be the exact parts or non-exact parts (e.g., using 4 spoons and a cuboid to assemble a table shape). The core idea is to use PointNet to extract per-point features in the input shape and per-part features among the input parts and then employ a multi-attention transformer across the shape points and parts first to perform part segmentation and then optimize for the output part poses. The authors compared the proposed method to several baseline methods using the PartNet parts and show better and more generalizable results than baselines. Results on real scans are also presented. Failure cases and limitations are well discussed.
review: Strengths:
- The paper presents a novel and interesting task of general part assembly.
- The proposed method is technically novel, specifically in proposing to do segmentation first and using optimization-based pose estimations for better generalization capabilities. 
- Experiments show clear advantages over baseline methods both qualitatively and quantitatively.

Weaknesses:
- It's a bit unclear why the proposed method would not overfit the training categories as the baselines do, since the PointNet/transformer backbones may also be subject to overfitting as the baseline methods. Or in other words, may the baseline methods also be able to generalize well to novel categories with increased numbers of the training categories? Can the authors show results on more test categories that are dissimilar to the training categories?
- The proposed method is similar to [3] which also proposed to do segmentation first but it's in 2D. It would be better to add a comparison to an adapted version of [3] to segment the input shapes in 3D.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, this is a good paper proposing a new interesting and important problem and presenting a technically novel method with good experimental results supporting its contributions. I'd like to see my questions addressed in the rebuttal, but they are minor points only serving as suggestions on potentially improving the paper.

Post rebuttal: Thanks for authors' rebuttal. I keep my score as Weak Accept.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: The paper formulates the general part assembly problem as a goal-conditioned shape rearrangement task, and proposes a Transformer-based architecture that processes input point clouds in a fine-to-coarse manner in two steps: predict a segmentation of the target shape and subsequently infer the pose of each part. Experiments on PartNet demonstrate the method and show its superior performance compared to baselines. Additionally, generalization to real-world scans is shown on the redwood dataset.
review: High quality paper, clearly written, original, and providing a significant step forward on the part assembly problem.

The evaluation is a bit limited, as the method was trained on "chairs, lamps, and faucets" and tested on "tables and displays". Is there a reason why only so few categories were used? What would happen if one swaps some categories between train and test sets (e.g., swap tables and chairs)?
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: A strong high-quality paper that proposes a general approach for solving part assembly problems on point clouds. In spite of the demonstrations being presented on a restricted set of categories, the results appear convincing and promising. Combined with task and motion planning, this method can contribute to a general autonomous robotic assembly system.
review_title: null

Meta-review: The paper introduces a novel approach to the problem of 3D shape part assembly planning by formulating it as a goal-conditioned shape rearrangement task and proposing a Transformer-based architecture. The proposed method shows promise in solving this task, and it has been received positively by reviewers, with both strengths and areas for improvement highlighted.
Strengths identified by reviewers include the quality of the paper, its clarity, and the originality of the approach. The method's technical innovation and potential for impact in the field have been highlighted. The paper also addresses limitations clearly and provides solid experimental results.
Reviewers have raised questions and suggestions regarding the evaluation setup. Specifically, concerns were raised about the limited number of categories used for testing and the potential effects of swapping training and testing categories. The authors have responded by explaining the limitations of available dataset categories and conducting a preliminary experiment to evaluate the impact of category choices.
Another point of discussion is the similarity of the proposed method to prior work in terms of generalizing to novel shapes. A related work by Funk et al. [1] was brought to the authors' attention, demonstrating the importance of acknowledging such related works and highlighting the differences between them.
The paper is considered to be of high quality, presenting a significant step forward in addressing the part assembly problem with promising results and innovation.
Post-rebuttal comments show that reviewers have maintained their recommendations, indicating that the authors' responses effectively addressed their concerns and reinforced the paper's value.
Decision: Accept (Oral)