Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a novel normal estimation method called HSurf-Net. The method works by implicitly learning hyper surfaces in a high dimensional feature space. The method includes a novel space transformation module consisting of a sequence of local aggregation layers and global shift layers, to reliably build the feature space. Experimental results show that the HSurf-Net can accurately predict normals from point clouds with noise and density variations, and achieves state-of-the-art performance on multiple datasets.
strengths_and_weaknesses: Strengths
- The paper is well-written and easy to follow. The method is technically sound. The authors have done abundant experiments covering a broad range of baselines and datasets, making the results convincing.
- The idea of transforming points into a higher dimensional feature space and fitting hyper surfaces is interesting and novel. The authors have designed the network accordingly to achieve this goal and performed ablation studies to validate each design choice.
- The method has shown impressive robustness towards noisy inputs. The HSurf-Net outperforms all other baselines under all settings.

Weaknesses
This paper is well-presented and proposes a novel model for normal estimation with state-of-the-art performance. I do not see any major issue in its current form. 
questions: - There are a lot of different notations (especially hyper-parameters) in this paper. Organizing them into a table in the appendix may help readers to follow the paper more easily.
- The idea of hierarchical point set abstraction (Fig.3) is actually proposed by PointNet++. The authors might want to cite this paper.
- Include a real-world example (e.g. a lidar scan) could further strengthen the paper.
- According to the theory proposed in this paper. The points are lifted to noise free features before fitting hyper surfaces. Could we utilize this feature to perform other point processing tasks (maybe jointly) e.g. denoising / super-resolution? Any discussions on this would be appreciated.
limitations: Yes, in the supplementary material.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: - This paper introduces an approach to surface normal estimation from the 3D point clouds. Instead of applying function fitting to the input data that may undergo overfitting or underfitting, the approach introduces hyper surface fitting to learn hyper surfaces in the high dimensional feature space implicitly. The approach learns hyper surfaces from the features and directly predicts normal vectors. The approach (HSurf-Net) is validated with the synthetic and real-world indoor and outdoor datasets.
strengths_and_weaknesses: **Strengths**

1. The paper revisits the normal estimation problem and considers the problem as the hyper-surface fitting problem. Since the surface fitting happens on the latent feature domain, such implicit fitting is rather robust to the explicit approaches that are easily affected by additive noise. The reviewer is not fully tracking the literature on this field, but it seems novel and interesting.
2. The paper comprehensively summarizes the related work in the normal estimation field. The paper is self-contained, so the readers readily follow the problem and the recent advances in this field.
3. The paper is straightforward to understand, and the approach shows compelling results on the PCPNet, SceneNN, and Semantic3D datasets. The visual quality of the estimated normal is reasonable.
4. The paper explains the proposed idea in detail. The supportive figures (such as Figures 2 and 3) help to understand the approach better.

**Weakness**
1. It is unclear that the proposed approach is completely free from ‘underfitting and overfitting'. The results may depend on the ‘c’ dimension of the hyper surface coefficients (Sec. 3.2). Even if the relevant features and network output would fit for the ‘c’, there is a danger that network capacity would be overparameterized. Can the authors mention this? This is an important issue because the main argument of the proposed approach is expressed in this manner. For instance, lines 7-9 “fitting surfaces explicitly from arw point clouds suffers from overfitting or underfitting issues caused by inappropriate polynomial orders and outliers… To address these issues, we introduce hyper surface fitting to implicitly learn hyper-surfaces…”
2. The relative position encoding module directly embeds the relative position in a local frame (p^j_j-p_i). However, such embedding would work for the specific metric scale. How would the approach handle the point clouds with non-metric scale point clouds? What I mean is that the scale set of the patch size {N, N/2, N/4} would be invariant regardless of the scale of the point cloud, but (p^j_j-p_i) would be directly affected by the scale of the point cloud. In other words, if there are two point clouds {X} and {X/10}, can the network would produce the same output?
3. It is interesting to see that Eq (9) induces the normal vector. However, the equation itself does not guarantee ||n||=1 because it is the network output of the max-pooled feature vectors. Does any additional post-processing is applied to get the normalized n? Please clarify.
4. One critical concern is whether the approach would handle uneven point clouds. The paper mentions the ‘patch’ over the paper, and mention that ‘patch size’ as the number of the points (Line224 and so on). However, such a definition of ‘patches’ is readily affected by the density of the point cloud. For instance, in the KITTI dataset, in the coarse region, if we set N=700, it will introduce severely biased point cloud samples. The all experiments are conducted with the evenly sampled point clouds, where the density of the point cloud is fixed over the dataset. Please state how the proposed approach would handle density changes or uneven point clouds.
5. In the experiment section, the paper states the proposed approach outperforms the previous approaches, but it would be better if the paper mentioned the comparison between the second-best approach. For instance, AdaFit[45] is quite similar to the proposed approach. Please provide some discussion or analysis of the suggested approach.

questions: 1. Figure 5 is hard to distinguish the proposed approach. Is there any way to visualize the errors more effectively? How about utilizing log-scale errors or changing the color map? Similarly, Figure 4 could be improved as well. Because many plots are overlapped, consider adjusting the min and max of PGP to show the other plots better. Consider using other line styles to distinguish the lines better.  
2. Another suggestion is to apply the estimated normal to the various applications. For instance, making a mesh from a raw point cloud remains a challenging problem. How about showing that the noisy raw point clouds can be successfully reconstructed as a mesh using the proposed approach? Maybe using the Poisson reconstruction using the various estimation of surface normal?
3. Please answer the questions in the paper weakness section.

limitations: - The paper does not address the limitation of the proposed approach. The concern about the network overparameterization and the issue about point cloud densities should be clarified and stated in the limitation section.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a method to estimate normals in point clouds based on a novel patch-based network architecture that integrates global patch information and local neighborhood information inside the patch in multiple steps, making heavy use of densenets and skip connections. A new cross-product-based loss is used to train the network.

I would consider the novel architecture, as well as the novel loss (which seems to be very relevant to achieve good performance according to the ablation) to be the main contributions.
strengths_and_weaknesses: Strengths:
- Normal estimation should be of great interest to the community.
- The architecture is reasonably novel. It roughly follows a PointNet-like setup in most components, but the component combination into the full architecture seems novel.
- The evaluation shows clear improvements over the state-of-the-art on multiple datasets, including real-world data.
- There is a thorough ablation of the design choices (even if their theoretical motivation provided by the authors is misleading, this provides some empirical motivation).
- The authors include source code and promise to provide their exact data.

Weaknesses:
- The motivation and presentation of the method is misleading.
- The exposition is missing some details of the method.
- Given that the current motivation is misleading, this leaves the design choices for the architecture poorly motivated.

Overall, due to the promising that show a clear improvement over the state-of-the-art on several datasets, I am leaning towards the positive side. However, the misleading exposition and missing details lower the quality of the paper significantly and should be corrected before the paper can be accepted. I believe that this can be done in a minor revision.

Details:
- The motivation and presentation of the method is misleading. In the introduction, Section 3.1 and Section 3.2, the approach is carefully motivated as surface fitting in a feature space. However it seems like surface fitting is not done explicitly, and instead a generic network is used that may or may not learn to do something similar as the polynomial surface fitting described in Eq. 5. The surface mathcal{N} is never explicitly constructed or encoded in any specific way in the network structure. Considering this I would clearly count the proposed method as regression-based method in the author's terminology, not as a hybrid or fitting-based method. The motivation of the approach as surface fitting seems to me like carefully motivating a method as specifically using a Sobel Filter to detect edges, and then training a generic CNN to obtain the output. It seems quite misleading and should be changed.

- The exposition is missing some details and could be improved.
  - In Section 3.1, the use of two Taylor expansions seems unnecessarily complicated. Also, since both are defined with the same order it does not become clear that the fitted surface can usually only be an approximation of the true surface. It might be clearer to define the ground truth surface with a generic function f(x,y) instead of the truncated Taylor expansion J_beta,n, and then fit J_alpha,n to this generic function.
  - The introduction should clearly mention that the goal is to find *unoriented* normals, i.e. without giving information which side of the surface is inside or outside.
  - In Eq. 5, the dimensionality of the feature space of F is not defined (the output space of mathcal{N}_theta,tau and mathcal{F})
  - The text below Eq. 5 mentions basis vectors that for G and C that are introduced in later sections, but I could not find mention of them in the rest of the text.
  - Eq. 6 is missing a vector norm for the difference between feature vectors.
  - Also, in Eq. 6, would optimizing over tau not always result in the highest possible tau being used? Since higher-order should always be able to achieve a lower error. (This may not be very relevant because no optimization is done in practice - see concerns about the motivation of the method.)
  - Section 3.2 should clarify how weights w_i are computed.
  - Losses should be discussed in the main paper, since they are non-standard and seem to be an essential part of the contribution. Shortening or changing Sections 3.1 and 3.2 could make space.
  - For Eq. 9, some motivation should be given for the design choice of using the max over all features, since this breaks the analogy to 3D surface fitting, where it would not make sense to take the max over all z values of the fitted surface.
  - In Section 3.3, it should be clarified if the kNN search for neighbors is performed in the input 3D space or the feature space. Also in Section 3.3, it should be clarified if the dense blocks are applied per-point or over the concatenated features of all grouped points in a neighborhood. If it is per-point, the local aggregation layer could be described more succinctly as a PointNet over the local neighborhood, with dense blocks instead of MLPs.
  - Section 3.3 is missing a description how G_s,i is converted to G_i. Is the last (coarsest) scale used? This should be clarified.
  - Section 3.3 and 3.4 are missing descriptions how the coarser subset of M points is chosen from the input N points. Farthest point sampling like in PointNet++? A random subset? This should be clarified.
  - In Section 3.4, the position encoding function phi is not defined. Is it an MLP as well? This should be clarified.
  - Section 3.4 is missing a description of how C^j_i is converted to C_i. Maxpooling? This should be clarified.
  - Section 4 should briefly mention how many point clouds were used from each dataset and the average point cloud size.
  - Section 4 should clarify if all baselines were trained on the same dataset.

- In the ablation, how was the MSE loss applied to the unoriented normals? This should be clarified (a naive application to output normals would not work since the normals are not oriented).

- The following papers could be added to the related work:
  - Stable and efficient differential estimators on oriented point clouds, Lejemble et al., SGP 2021
  - PCT: Point Cloud Transformer, Guo et al., CVM 2021
  - Point Transformer, Zhao et al., CVPR 2021 (they don't demonstrate normal estimation, but their architecture is for general point cloud processing)
questions: The authors should clearly state in the rebuttal if they plan to change the motivation of their method as described above, or alternatively if I missed something that justifies the current motivation given by the authors.
limitations: The authors have sufficiently addressed limitations and societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new method for the task of normal estimation for point clouds using hyper-surface fitting. The model consists of relative positional encoding, and a space transformation module (MLP) to map 3D point clouds into feature space where the hyper-surfaces are fitted. The method is evaluated on three different datasets, including PCPNet, ScenNN, and Semantic 3D and shows state-of-the-art performance in RMSE and PGP metrics. 
strengths_and_weaknesses: The paper is mostly well written and easy to follow. The method is useful as shown by the results on multiple datasets. Different elements of the model are well analysed by an ablation study and show the contribution of each of the components. Figures 1 and 2 help the reader follow along and clarify the overall framework. The supplemental material also contains interesting results. 
questions: There are several aspects that I am not clear on: 
1. The space transformation module is unclear to me and I found Figure 3 confusing. How is it different from a PointNet++ idea? While clearly, the architecture is different the concept of multi-scale and grouping was first proposed there, so how is this different? 
2. Novelty: To my understanding, this method could be interpreted as a newer version of PCPNet with PointNet++ as backbone architecture and Positional encoding as input. Each has already been proposed in previous works and showed its effectiveness and the question is how is combining these building blocks novel? 
3. Insight: The main paper lacks insight into what the network is actually learning. There is some of it in the supplemental but not sure that it is enough. It would be interesting to visualize each point's contribution to the hypersurface fitting as a heatmap. 
4. Will the code be released ? 

limitations: The limitations are not clearly presented. Clearly, one limitation is robustness to noise. What are other limitations? When does it fail in the clean point cloud cases? Additionally, no potential societal impact was mentioned (since it is a normal estimation method, I am not sure there are any but even if there aren't, it should at least be mentioned).
review_title: null

Meta-review: metareview: This paper proposes an approach to fit implicit surfaces for surface normal estimation. Reviewers unanimously agree on its novelty and performance. AC hence recommends acceptance.
confidence: Certain
Decision: Accept