Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a novel unsupervised framework for representing repeatable parts as shape programs with part-based implicit functions. The authors propose to utilize implicit functions to represent parts, which improve the representation capacity of shape programs. Meanwhile, a matching-based unsupervised learning objective is devised to learn the shape program from unannotated shapes. Experimental results show the effectiveness of the proposed method in shape reconstruction fidelity and semantic segmentation for chairs, tables and airplanes of the ShapeNet dataset. 
strengths_and_weaknesses: Strengths: 
+ Both the problem addressed in the paper and its proposed solutions are innovative and interesting.
+ The proposed matching-based unsupervised training objective is sound.
+ The experiment shows quite marginal performance improvement.
+ The paper is well-organized and easy to follow.

Weaknesses:
I don't really have any major concerns; some minor concerns are as follows:  
1. Since part-based implicit shape representation is used, it would be better to compare the training and testing complexity between the proposed method and the baseline, such as BSP-NET.
2. Does the proposed method require a canonical orientation of shapes.
3. the authors should provide qualitative and quantitative results on more categories, such as car and lamp.
4. It would be good if the authors could conduct more experiments to show the expressiveness of the new shape representation. For example:
i) It would be interesting to explore the feature space to validate if the method can accomplish smooth interpolation and extrapolation of shapes. 
ii) Can the new shape programs be adapted for single-view 3D reconstruction?
5. Are the trained models category-specific or a single universal model is trained for these 3 categories? How do you make sure the comparison is fair in Tab. 1. All baselines are trained in the same experimental setting.


questions: Please see the weaknesses.
limitations: Please see the weaknesses.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a method to reconstruct part-based shape models from point clouds, where each part has a local neural representation of the part's geometry. Additionally, multiple instances of the same geometry can be re-used by different parts, thereby explicitly modelling translational and rotational symmetries. The point cloud is encoded into a global feature vector, that is then used in two transformer-based steps to first generate part geometries and then translated and rotated instances of the part geometries. The method needs a part decomposition of the training shapes as supervision, although this decomposition can be obtained using existing unsupervised methods. A novel loss allows for efficient training of parts without requiring explicit correspondences with the ground truth parts. The authors show that this approach allows for part-aware shape reconstruction, unsupervised segmentation, and shape editing.

The two-step generation approach with geometries and instances and the loss with shape-based part assignments both seem like interesting contributions to me. Using neural representations of part geometries seems a bit less novel, since it has been used before in prior works (for example, Neural Parts, Local Deep Implicit Functions).
strengths_and_weaknesses: Strengths:
- The separation of part generation into geometry and instances seems like an interesting idea to make use of the compositional nature of many shapes, and seems to be relatively novel in the context of part-based shape generation.
- Implementation seems technically solid.
- The paper is generally well-written and easy to read.
- The evaluation shows some advantages in unsupervised shape segmentation.

Weaknesses:
- The authors do not clearly demonstrate the advantages of their contributions. What is the advantage of having a representation with repeatable parts over current work with non-repeatable parts or current work without parts? An more thorough evaluation of some applications that are only possible with repeatable parts, or that clearly perform better than the state of the art would be necessary.
- The evaluation focuses on shape reconstruction, however existing methods like Occupancy Networks perform significantly better on shape reconstruction (Occupancy Networks should be included in the left part of Table 1). There could be other applications where working with repeatable parts is necesary or beneficial (like part editing, part mixing, inductive biases for generation, interpolation, etc.), but these are not demonstrated thoroughly enough. Only one example of shape editing is provided.
- The segmentation experiments are reasonable, but by themselves not sufficient for acceptance, and Table 1 in the supplementary seems to suggest that using repeatable parts actually hurts segmentation performance compared to using non-repeatable parts.

In summary, I like the idea of using repeatable parts to represent shapes, but the evaluation does currently not convincingly demonstrate the advantage of such a representation.

Details:

- Using a part-based representation with part re-use is described as a major advantage of the proposed method in the introduction, and it is a major part of the technical contribution. However, from the evaluation it is unclear what the advantage of such a representation is over the state-of-the-art. For reconstruction accuracy, non-part based representations like OccNet seem to perform better, so that does not seem to be a strong suit of the proposed representation, or possibly of part-based representations in general. The segmentation comparison does show some advantage in Table 1, even if the results are a bit mixed, but Table 1 of the supplementary shows that the segmentation performance is lower with part re-use. There is one edit example in the supplementary that shows some down-stream advantages of part re-use, but additional examples, ideally with more repeated parts (storage furniture?) and possibly mirrored parts (usually the chair arms are mirrored, not related by only translations and rotations) would be more convincing. Also, metrics that specifically measure the success of the part re-use are missing, for example measuring how often symmetric parts in a shape are correctly modeled with the same geometry and how often parts that have different geometry are incorrectly modeled as instances of the same geometry.

- Comparisons to Neural Parts and Local Deep Implicit Functions (LDIF) both seem relevant, since they both perform unsupervised shape decomposition and use neural representations of part geometries. For LDIF, the authors argue that it is not included due to using a local feature encoding compared to the global feature encoding of ProGRIP. What are the specific applications or tasks that LDIF can't do due to its local feature encoding, but that ProGRIP can do? This needs to be discussed and clarified if the goal is to show that an empirical comparisons to LDIF is not needed since there is a clear theoretical advantages. I can imagine, for example, that a single global latent space enables interpolation or generation, which may be more difficult with local latent spaces. Although ideally such an argument is backed up by empirical evidence.

- It might be good to clarify the extent to which the method is unsupervised in the introduction. It is true that the method can be trained on an unannotated dataset, but only if a part annotation is generated for the dataset as a pre-processing step, possibly using existing unsupervised methods. Calling it a fully unsupervised approach in the introduction without further clarification may incorrectly create the expectation that the method is trained end-to-end with only the final occupancy loss as supervision.

- To a lesser extent, calling the representations a shape program may also create the unrealistic expectation of working more complex program. The current representation may better be described as structural representation rather than a program, since only two types of 'operators' are effectively used, always in the same order: first box creation and then instancing. But I would consider fixing this point (for example by changing the title and introduction) optional.

- I agree with the authors that representation compactness is a good measure for a learning-based model. However, it seems like the F-score the authors use measures the reconstruction quality with geometry (the information stored in the z-vectors), but only measures space requirements without geometry (i.e. without counting the information stored in the z-vectors). This seems like an unfair comparison, as the other methods do not make use the z-vectors to store additional information that is useful in improving the F-score.

- The following papers could be added to the related work:
	- SPAGHETTI: Editing Implicit Shapes Through Part Aware Generation, Hertz et al., ArXiv 2022 (concurrent work)
	- SDM-NET: Deep generative network for structured deformable mesh, Yang et al., Siggraph Asia 2019
	- PQ-NET: AGenerative Part Seq2Seq Network for 3D Shapes, Wu et al., CVPR 2020
	- DSG-Net: Learning Disentangled Structure and Geometry for 3D Shape Generation, Yang et al., Siggraph 2022 (concurrent work)
	- Generative 3D Part Assembly via Dynamic Graph Learning, Huang et al., NeurIPS 2020
	- Write, Execute, Assess: Program Synthesis with a REPL, Ellis et al., NeurIPS 2019
	- InverseCSG: Automatic Conversion of 3D Models to CSG Trees, Du et al., TOG 2018
	- Engineering Sketch Generation for Computer-Aided Design, Willis et al., CVPR 2021Workshop paper
	- Learning Adaptive Hierarchical Cuboid Abstractions of 3D Shape Collections, Sun et al., Siggraph Asia 2019
	- ParSeNet: A Parametric Surface Fitting Network for 3D Point Clouds, Sharma et al., ECCV 2020

- Since the box generation model is pre-trained without knowledge of the geometry, it should create two instances of the same box in cases where two boxes are have the same shape. But same box shape does not necessarily mean same geometry (for example, two boxes may contain mirrored geometry, or just geometry that has slightly different details, etc.). Did you observe the model successfully splitting two instances of the same box into two different boxes in the refinement step if the two boxes are the same but not their geometry? Showing such an example might be good to confirm that this is not a problem.

- Transformers typically work on ordered sequences (step i takes the output of step i-1 as input), some details should be given how the transformer is set up here to be order-invariant. For example, does each step still output a probability distribution that is then sampled probabilistically (the standard setup for transformers), or does each step regress/classify output values deterministically (given the input)? If each step samples a potentially multi-modal probability distribution, how are the samples of different steps coordinated to make sure only compatible modes are selected when sampling?
 
- Allowing for a reflection transformation in the instances seems like it could enable significantly more part re-use, since parts are often mirrored. Was there a specific reason this was not used? Or can the rotation R also be used to model a reflection? This could be discussed in the future work for example, or a clarification is needed that R can also describe reflections.

- The threshold used to compute the F-score metric should be mentioned.
questions: Providing a clear description of what the authors consider to be the main practical contributions of their representation with repeatable parts of existing shape generation methods, and how these contributions are demonstrated in the paper would be good.
limitations: The authors show some limitations of their method.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new method for geometric and structure reconstruction of composite 3D shapes. The main contribution consist of automatically detecting and encoding the repeated parts. This leads to a relevant contribution in comparison with existing work. Moreover, the single parts are accurately reconstracted using neural implicit function. The proposed approach is fully unsupervised. An exhoustive experimental section shown the improvement of the proposed approach in comparison with other methods for both shape recontruction and shape segmentation. 
strengths_and_weaknesses: *Strengths*
-the paper focuses on a relevant problem that is the automatic encoding of composite objects. The capability of detecting repeatable parts is very important bringing a significative step toward the semantic understanding of objects and scenes. 
-the use of deep implicit function enables the method to reconstruct more accurate details of the observed object improving the geometric representation of the observed parts  (differently than other method that capture only a coarse approximation of shape such as 3D boxes).
-the proposed composite encoding pipeline is well designed by properly combine existing methods with novel parts. 
-  experiments are well organized and shown promising results in comparison with other methods.  

*Weaknesses*
-It seems that the proposed work strongly believes on the results of [55]. In this sense the proposed work can be considered as an extension or a refinement of [55]. Authors should better explain this point.
-The proposed approach seems computational demanding, authors should discuss and evaluate the computational complexity of their work.
-The learning part is not fully novel since the main parts are based on alredy available neural architectures (for box estimation [55], and implicit function estimation [33, 38, 7]).

questions: -A better explanation on the relation between the proposed approach and [55] is required to understand the impact of the proposed method (i.e., what happen if the method [55] fails).
-What is the computational complexity of the work in comparison with other methods? 
-Authors emphasized that they proposed a shape program approach but it seems that  ‘program’ part is not considered in the methodological section and results. Authors should clarify why they consider their work as ‘program’-like (or clarify better the meaning of shape program).  

Minor points:
Is it possible that scale s_i change for the same part? (i.e., two reapeted parts with different scales).
It is not clear the meaning of NM on line 193. Authors wrote that primitives are estimated using [55], but NM is predicted by ProGRIP, please clarify. 

limitations: Limitations are not discussed. For instance authors should declare their strong dependency from [55].
review_title: null

Meta-review: metareview: All reviewers recommend acceptance of this paper. They find the approach of repeatable parts innovative and the paper well written. The AC concurs
confidence: Certain
Decision: Accept