Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a 3D Shape reconstruction method using local patch priors. The proposed method uses the multi-resolution patch priors based on the observation that within a 3D structure, some details are repetitive and often easier to be constructed first. The paper is well written and balanced 
strengths_and_weaknesses: + the presentation of the paper is well put together

+ the illustration of the paper is detailed and easy to understand

+ the idea of using local prior and multi-resolution is not necessarily brand new, but it is effective for the method the paper is proposing.

+ based on the paper's experiment results, it indeed improved some real-world 3D reconstruction

- it would be great if the author could showcase some reconstruction from the real world without a ground-truth scan. Just a static RGB photo and the 3D reconstructed object would help better demonstrate the strength of the proposed method.
questions: * the method is indeed effective over the test case, how about the efficiency? what is the average runtime compared to other signal-pass and one-resolution methods?
limitations: Natural limitations as the author also pointed out, the bounding boxes are required for the scan, which means this method is constrained to a predefined space and is not directly translatable to use in the wild.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a deep network architecture for performing 3D shape completion on unseen shape categories by learning local geometric priors at multiple scales. By learning sets of features for distinct categories on patches sampled at different resolutions and fusing these features with those computed from an input partial scan from an unseen category, the network is able to leverage the local information  to achieve a better reconstruction. The method is validated qualitatively and quantitatively against state-of-the-art baselines, and a brief ablation study is provided to justify the multi-resolution aspect.
strengths_and_weaknesses: This paper makes a nice step towards training shape reconstruction models that can handle out-of-distribution examples from unseen categories. The multi-scale approach is sensible, and the experimental results validate its effect.

The results that are shown are all quite low frequency, with the shapes having fairy limited local variation. I wonder why this method is unable to handle more detailed geometries. Is this due to the fact that such textures are not seen in the training categories, or is there a different bottleneck?

The method description is pretty heavy on notation and somewhat difficult to follow. It would be helpful to clearly define the different objects that are considered and clearly distinguish between geometry representations and latent high-dimensional encodings. To this end it's also important to fix subtle typos in the notation, e.g. $S$ should be $\mathcal S$ on L107.
questions: Why is the softmax denominator in (1) $d/2$? This seems to depart from the usual $\sqrt d$.

How are the partial shapes inputted to the pipelines as SDFs? Aren't the partial shapes generically not closed surfaces?

Why do the ground truth synthetic surfaces in Figure 5 have artifacts and imperfections?

To what extent is the method robust to some of the hyperparameters used? For example, in particular how important is the choice of resolutions and the number of layers in the pyramid in the multi-scale approach? Similarly, how are the coefficients for the loss terms chosen?

How is the clustering to initialize the shape priors performed? Is it done directly on the SDF representations?
limitations: Limitations are sufficiently discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles a challenging problem - the performance degradation on unseen categories in shape completion tasks. Inspired by the observation that different categories may share similar local geometric structures, the authors propose a method to learn multi-resolution patch priors and then reconstruct the complete shape using the recursively fused features. The results show a significant performance improvement in novel-category shape completion compared to baseline methods, and ablations have verified the importance of each component.
strengths_and_weaknesses: Learning patch-level priors and the fusion pipeline of different resolution patch priors is moderately novel to me. The pipeline in this submission is technically sound. The submission is clearly written and well organized. The authors also provide the source code in supplementary material for better clarity of the techincal details. BTW, I’m a bit impressed that the completed shapes are mostly watertight and have continuous geometry. 

In general, the evaluation part of the manuscript is good to me. The numerical results show a significant performance improvement over the baseline methods on both synthetic and real scan data. However, as the authors mentioned in limitations, visual and qualitative results on categories with fine geometric structures are not shown, which could have been a strong support to the merit of the proposed method. For the ablation study, the authors answer the importance of each designed component and show convincing numerical results of the effectiveness.


questions: 1. I wonder about the criteria for splitting the datasets. In my opinion, the partition of the data has an enormous impact on performance (e.g., it is hard to complete a chair if we only have sofas as priors). If you randomly choose the training categories, I think it would be better to use cross-validation to show the impact of how the data is divided or the generalization performance. 

2. Most of the categories in the paper and supplementary only have simple geometry, such as bed or bathtub. Although you mention the lack of fine-scale geometric details in the results, I am still curious about performance on more complex data (such as the chair or bookshelf you mentioned in the video).

3. What is the impact of the number of priors? In the ablation part, the authors show the results on different patch resolutions, and it seems only 4^3 priors can achieve competitive performance. Can I assume that local path priors play a major role, and it may have less diversity? So it may have redundancy, and the boundary of the number of priors is meaningful in real-world applications. Moreover, it would be better to show the memory usage for storing the priors.

4. How about the domain independence of the proposed method? Have you tried the cross-dataset experiments? More specifically, can you train priors on ShapeNet, and test using ScanNet? Logically speaking, there should be a reasonable result.
5. By the way, why choose TSDF as the input shape representation? Is it because it is easy to perform 3D convolution operations? Is it possible to change TSDF to point cloud?
limitations: Tthe paper bills itself that it can complete shapes for entirely unseen categories, but the authors do not illustrate the criteria of splitting the categories in the dataset, making the generalization of the proposed method on unseen categories obscure. It would be better to show more results on this to support the claim. 

Another concern is the performance on categories with more complex geometries, logically, learning over the patch level could be helpful to improve the local geometric details in the synthesized results. Although the authors mention it in the limitations, the results of the method in its current form do not fully show the power of leveraging patch-level priors.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The submission introduces PatchComplete for 3D object-level shape completion from a partial input SDF voxel grid, with a focus on generalizing to unseen object categories (not just unseen instances). It evaluates on the synthetic ShapeNet and the real-world ScanNet, which contains 3D reconstructions of rooms (ScanNet annotations are used to segment out objects; pseudo-GT full object shapes are obtained via Scan2CAD). The paper contributes a scheme to learn patch-level shape priors. It also contributes a method that associates an input incomplete shape with that patch-level prior and then reconstructs the complete shape as a (learned) merging of that associated/weighted prior information. Comparisons are done to a diverse set of prior work and PatchComplete outperforms them, especially on unseen categories. A number of ablations are also included.
strengths_and_weaknesses: _Positives_

The method is novel and enables good generalization to unseen categories, which is particularly useful for static 3D reconstruction of unseen environments with systems like KinectFusion.

The paper is written well in terms of language and clarity. 

The experimental evaluation is done well. The ablations consider a multitude of factors of the method.

_Negatives_

I don't see much on the technical level that could potentially be used in other areas of CV/ML. The method consists of a core (attention into a learned patch prior) and a number of tweaks, which are evaluated in the ablations. The core part is a simple version of (non-sparse) dictionary learning. 
questions: Questions for the rebuttal:

1) It is not clear to me whether the accompanying code will be released.

2) I have several questions about Table 3:
a) The ablation of which resolutions to use in the multi-resolution scheme (Table 3) shows rather marginal improvements when going from only the finest resolution (4^3) to all three resolutions (Ours; 4^3, 8^3, 32^3). Are really both 8^3 AND 32^3 necessary? How does 4^3+8^3 and 4^3+32^3 perform? 
b) What are the standard deviations for these numbers? Are the marginal improvements significant? 
c) Especially on ScanNet, which is the more interesting dataset compared to ShapeNet for this task, shows results that are on par for 4^3 only and for Ours. I am not really convinced that this supports the claim that using multi-resolution leads to the "most effective shape completion results" (line 216).

3) An ablation of the concatenation in Eq. 3 could be added, to get a sense of how much the patch prior and how much the input encoding can achieve on their own, and what their respective issues are. I.e., use only Q^R_i in one ablation and use only the Attention() in another ablation. This is not crucial though.

4) What is the "no attention" ablation in Table 4? What is used instead of attention? Only Q^R_i directly, as I suggested in 3)? What else is changed?

5) An overall ablation that uses fixed priors, uses no pre-training, uses no attention, and only uses resolution 4^3 would be interesting. How much do these tweaks contribute to the overall performance?

Suggestions for improvement that are not relevant for the rebuttal:

- I suggest to add qualitative results for the ablations, at least in the supplement. 

- How many shapes/priors does {T^c} contain? I.e. how many representative samples (in total) are there out of how many total shapes {G^c}?

- Some related works on deep implicit local geometry representations could be added:

* Genova et al. Local Deep Implicit Functions for 3D Shape, CVPR 2020
* Charbra et al. Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction, ECCV 2020
* Takikawa et al. Neural Geometric Level of Detail: Real-Time Rendering With Implicit 3D Shapes, CVPR 2021
* Jiang et al. Local Implicit Grid Representations for 3D Scenes, CVPR 2020
* Deng et al. NASA: Neural Articulated Shape Approximation, ECCV 2020

- There's a superfluous period in line 95.
limitations: - Limitations are mentioned. 

- Societal impact is discussed.
review_title: null

Meta-review: metareview: This is an interesting paper on class-independent 3d shape completion. Reviewers agree that the paper has good quality and is moderately original. There were initially some questions about the level of generalization to new classes, but after a strong rebuttal all reviewers find the results compelling and all of them suggest acceptance. I agree with their assessment.
confidence: Certain
Decision: Accept