HSurf-Net: Normal Estimation for 3D Point Clouds by Learning Hyper Surfaces
Authors: Qing Li
Yu-Shen Liu
Jin-San Cheng
Cheng Wang
Yi Fang
Zhizhong Han
Venue: NeurIPS 2022 Accept (2022)
Keywords: Point Cloud
Normal Estimation
Hyper Surface
Surface Fitting

Abstract: We propose a novel normal estimation method called HSurf-Net, which can accurately predict normals from point clouds with noise and density variations. Previous methods focus on learning point weights to fit neighborhoods into a geometric surface approximated by a polynomial function with a predefined order, based on which normals are estimated. However, fitting surfaces explicitly from raw point clouds suffers from overfitting or underfitting issues caused by inappropriate polynomial orders and outliers, which significantly limits the performance of existing methods. To address these issues, we introduce hyper surface fitting to implicitly learn hyper surfaces, which are represented by multi-layer perceptron (MLP) layers that take point features as input and output surface patterns in a high dimensional feature space. We introduce a novel space transformation module, which consists of a sequence of local aggregation layers and global shift layers, to learn an optimal feature space, and a relative position encoding module to effectively convert point clouds into the learned feature space. Our model learns hyper surfaces from the noise-less features and directly predicts normal vectors. We jointly optimize the MLP weights and module parameters in a data-driven manner to make the model adaptively find the most suitable surface pattern for various points. Experimental results show that our HSurf-Net achieves the state-of-the-art performance on the synthetic shape dataset, the real-world indoor and outdoor scene datasets. The code, data and pretrained models are publicly available.

HSurf-Net: Normal Estimation for 3D Point Clouds by Learning Hyper Surfaces
Qing Li' Yu-Shen Liu Jin-San Cheng?
Cheng Wang?
Yi Fang' Zhizhong Han
School of Software, BNRist, Tsinghua University, Beijing, China
Accadmyy of Mathematics and System Sciences, Chinese Academy of Sciences, Beijing, China "School of Informatics, Xiamen University, Xiamen, China
"Center for Artificial Intelligence and Robotics, New York University Abu Dhabi, Abu Dhabi, UAE
Departmennt of Computer Science, Wayne State University, Detroit, USA fleoqli, liuyushen)ee Stsiengunttnnnnaat.aa edu. cn jcheng@amss ac. cwang@xmu. edu. yfang@nyu. h312h@wayne.cn
Abstract
We propose novel normal estimation method called HSurf-Net, which can accu- rately predict normals from point clouds with noise and density variations. Previous methods focus on learning point weights fit neighborhoods into a geometric surface approximated by polynomial function with a predefined order, based on which normals are estimated. However, fitting surfaces explicitly from raw point clouds suffers from overfitting or underfitting issues caused by inappropri- ate polynomial orders and outliers, which significantly limits the performance of existing methods. To address these issues, we introduce hyper surface fitting to implicitly learn hyper surfaces, which are represented by multi-layer perceptron (MLP) layers that take point features as input and output surface patterns in high dimensional feature space. We introduce a novel space transformation module, which consists of sequence of local aggregation layers and global shift layers, learn an optimal feature space, and a relative position encoding module to effec- tively convert point clouds into the learned feature space. Our model learns hyper surfaces from the noise-less features and directly predicts normal vectors. We jointly optimize the MIP weights and module parameters in data-driven manner to make the model adaptively find the most suitable surface pattern for various points. Experimental results show that HSurf-Net achieves the state-of-the-art performance on the synthetic shape dataset, the real-worid indoor and outdoor scene datasets. The code, data and pretrained models are publicly available
Introduction
Estimating normals from point clouds is vital for various downstream applications of 3D computer
vision, such as point cloud filtering 5 46 35 34 surface reconstruction (77 and rendering [8 [5] 40]. Though this topic has been extensively studied, it is still a challenge to work on point clouds with different types of noise, outliers, and density variations. estimation can be formulated as the least squares optimization problem [12), which explicitly fits
well-known that point cloud normal
geometric surface (e.g. plane and polynomial surface) on local neighboring points and then computes the normal from the fitted surface. Specifically, the point-wise weights dedicate the importance of each point for the surface fitting.
Cooreespondig author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) Prior SOTAs
3D Point Cloud
Network
Weight
Geometric Surface Fitling
Surface Normal Estimation
Ground-truth
Outliers
Janla 6
(g)D Surface Finting Caves
3D Point Cloud
Space Transformation
Hyper Surface Fittnng ()
Direct Normal Estimaion
Underfitting
Overfitting
(b) Ours
Figure 1: (a) Prior SOTA methods focus on weight learning and geometric surface fitting to estimate the surface normal. (b) We use features C' to learn a hyper surface Ne.r Ne,r(G, C') directly estimate normal for each point. (c) Existing surface fitting techniques are severely affected by overfitting, underfitting and outliers, which lead to inaccurate surface approximation and normal estimation.
Existing normal estimation algorithms can be roughly divided into two categories: traditional methods and learning-basse -bandd methods. The traditional ones usually approximate potential structural properties of point clouds and use well-designed algorithm to fit local planes or polynomial surfaces [21 12]
However, explicitly fitting geometric surfaces heavily rely on careful parameters tuning, such as the neighborhood size and the order of polynomial function. The learning-based ones initially try
to directly predict normal vectors from point clouds through regression network [10 17
53 52].
Alternatively, recent methods employ Convolutional Neural Networks (CNN) [30] or the PointNet architecture [6 [55] to learn point-wise weights, then the classic geometric surface fitting is utilized to compute normals (see Fig. a The learning- based regression methods lack geometric prior and are hard to learn general mapping from severely degraded inputs to the ground- normals, especially for complex geometric structures. For the learning-basee surface fitting methods, one issue of these approaches is that explicit surface fitting is sensitive to noise and outliers. The weights on noisy points that are far away from the underlying surface significantly affect the accuracy of normals, even small weights on outliers still result in an erroneous normal estimation. Another inherent issue is that their predefined polynomial functions may not be suitable fit various surfaces since a constant order of polynomial functions is selected for all points, e.g. plane in [30] [T] and 3-jet surface in 66 55). If the selected order is smaller than the order of the underlying surface, will result in an underfitting, which smooths out the fine details and affects the accuracy of output normals. Otherwise, overfitting makes the algorithm sensitive to noise and brings instability to the normal estimation [55] as illustrated in Fig. [c).
To address these issues, we propose a novel network called HSurf-Net for unoriented normal esti- mation. It makes full use of the powerful learning ability of the neural network to implicitly learn hyper surfaces. The hyper surfaces are represented by MLP layers whose parameters interpret the geometric structures in a high dimensional feature space. The advantage of our hyper surfaces is to adaptively fit more complex point patterns in robust way. Based on the learned hyper surfaces, we introduce Hyper Surface Fitting module to directly predict normal vectors (see Fig. [b)). This module learns from the well extracted point features and gets the surface representation parameters optimized in a data-driven manner, rather than explicitly fitting 3D planes/surfaces by polynomial function with predefined order in current methods. Moreover, to avoid the selection of neighbor- hood scale and construct noise-less feature space, we design two network modules called Space Transformation and Relative Position Encoding. They can cover local, small and large scales to enhance the extraction of structure-aware and multi-scale features. Overall, the combination of these modules extracts the discriminative geometric information and avoids the issues caused by explicit polynomial surface fitting, thus improving the performance of the normal estimation framework. We conduct evaluation experiments on the synthetic shape dataset, the real-world indoor and outdoor scene datasets. HSurf-Net significantly outperforms other baselines the challenging cases in these benchmarks, and also shows a strong generalization capability on real-world LiDAR data. Extensive ablation experiments validate the effectiveness of each component that contributes to the final results.
Our main contributions can be summarized as follows.
A technique for representing polynomial surfaces as hyper surfaces, which are parameterized by MLP layers.
A Hyper Surface Fitting in a high dimensional feature space to optimize the surface representation for point cloud normal estimation, which brings more robustness and higher accuracy.

---

A Space Transformation module and Relative Position Encoding module to map 3D point clouds into the feature space. Their combination can fully explore the local geometry and extract features from different neighhorhood scales.
2 Related Work
Traditional Normal Estimation. The classic point cloud normal estimation methods use plane fitting techniques, such as unweighted Principle Component Analysis (PCA) [2]] and Singular Value Decomposition (SVD) 44 They analyze the covariance matrix in a local patch around a point and define its normal as the eigenvector corresponding to the smallest eigenvalue. Subsequently, variety 38 28 [24] have been proposed to improve the accuracy. In order to preserve sharp features and retain more details, lot of improvements were made to extract normals by using Voronoi cells [3 37]. Voronoi-PCA 14 2). Hough transform [9] and edge-aware sampling [25] In addition, some methods utilize more complex surface reconstruction techniques, such as Moving Least Squares (MLS) [31]. jet fitting [12], local spherical fitting [16] and multi-scale kernel [4]. The aforementioned approaches come with certain assumptions or specific observations, and hold theoretical guarantees on approximation and robustness. Besides, these approaches need a fine-tuned set of parameters according to the input point clouds.
of PCA-variants [1 39
Learning-based Normal Estimation. (1) Regression based methods. Initially, some learning- based methods propose to directly regress normal vectors from raw point clouds. HoughCNN [IO] transforms 3D points into a 2D grid representation via Hough transform, and then use a CNN to select normal direction from the accumulator in Hough space. Similarly, Roveri el al. 43 define a grid-like regular input to learn point normals. Lu et al. [33 project each point into 2D height map by computing the distances between points and the PCA fitted plane. These methods map the unstructured point cloud data into a regular domain, which may lose or alter the 3D geometric information. Instead, the following approaches learn from raw point clouds. PCPNet 17 applies the
PointNet architecture [4] in local patch-based multi-scale form to estimate normals and curvatures from point clouds. Hashimoto al. (00 propose a two-branch network that extracts local and spatial features, which are integrated to learn normals. Nesti-Net ? proposes to parameterize the local field as a multi-scale feature vector, then uses a mixture-of-eexprts architecture (26) to find the optimal neighborhood scale around each point, but it is very time consuming. Based on PCPNet, Zhou et al. [ introduce an extra plane feature constraint and multi-scale neighborhood selection strategy to improve the performance. Refine-Net [2] first computes initial normal at each point by searching an ideal fitting patch. Then, a refinement network is used to obtain the final optimal normals by incorporating the learned local point features and the constructed height map features.
(2) Surface fitting based methods. Recent normal estimation methods try to combine traditional plane/surface fitting techniques with learning- based methods. Lenssen et al. 30 propose fast algorithm that utilizes an adaptive anisotropic kernel to iteratively refine a weighted least squares plane fitting. MTRNet [1] proposes differentiable RANSAC-like module to predict a latent tangent plane. DeepFit 6 and AdaFit (55) both employ a Taylor expansion to describe the local surface and use the PointNet to predict point-wise weights for weighted least squares surface fitting. In addition, AdaFit proposes to use a novel layer to aggregate features from multiple neighborhood sizes and add offset to point coordinates to further improve the results. Similarly, Zhou et al. (54) use the learned weights and features to select local top-k points and update the point positions. Zhang et al. [49] introduce geometric weight guidance to provide more reliable inlier points for plane fitting.
Compared to the learning-based normal regression methods, e.g. PCPNet 17), Nesti-Net 7 and Refine-Net [52), which focus learning a general mapping from the inputs the ground-truth normals, our approach improves the normal estimation performance by deploying a hyper surface fitting to explore geometric prior supports in high dimensional space. Compared the learning- based surface fitting methods, e.g. DeepFit [6] and AdaFit (55), which solve a 3D space based polynomial function with a predefined order get normals, we effectively improve the results by learning hyper surfaces in a feature space.
3 Method
Given a local point set P (prli=1,,... N) centralized a query point p. our algorithm aims to estimate the unoriented normal np of point p. Fig. shows overview of the proposed approach.
3

---

Layer
Element- wise Multiply
Maxpooling
Space
Transfor mation
Concat
Local Palch
M-128
Ps
A
W
Ps
C2
Hyper Surface
f2
12
Fitting
Pt
Pioe
6x
Output Module
Query
Point Cloud
N-3
M-N
(4K)(3+64)
M-12B
Normal Vector
Space Transformation
Hyper Surface Fitting
x
x 3
Repest, 8t
X
x 5
N-128
M-12B
Fi2
91
M-128
Dense Black
M-K+(3+54
Local Aggregation Layer
Global Shift Layer
Residual Block
Figure 2: The architecture of HSurf-Net for point cloud normal estimation.
First, remove unnecessary degrees of freedom from the input data space and lower the learning difficulty, we normalize each point coordinate with its patch radius and rotate the points into local coordinate system defined by the PCA. Then, the Space Transformation module extracts global location code G for each point (Sec. 3.3). Moreover, local frames are formulated at point p; based on spatial coordinates and are used compute condition code C by a Relative Position Encoding module (Sec. 3.4). After that, we perform the hyper surface fitting in a high dimensional feature space (Sec. Finally, we recover the 3D normal vectors from fitting results by an Output Module.
3.1 Preliminary
We first briefly review the formulas and mathematical notations of the explicit surface fitting with polynomial function with predefined order. A smooth surface can be locally formulated as the graph of a bivariate height function fær y) about the z-axis that is not in the tangent space [4 29 An n-order Taylor expansion of the height function over surface is given by
12
f(x, u) (x, v) + (\)++)+)*), where Jg,n(x, v)
Bk
A=
The polynomial function Jz,n 2 RR the truncated Taylor expansion called n-jet 12 , the n-jet coefficient vector and involves N₂, (n + 1)(n + 2)/2 terms. O(-) denotes the remainder in Taylor's multivariate formula. The predefined order determines the complexity of the fitted surface.
Consider a collection of points P = fp; (z,, smooth surface S, and the height function = f v) given by Eq. ① in the defined coordinate system, we can investigate the n-order differential property of the surface S S= fx; Vs) at point p; by interpolating S using a bivariate n-jet (2, y), such that
NJ,,...,, N} around its origin on a sampled
(2) where a means the coefficient of the jet sought. Thus, the coefficient the solution of the n-jet surface fitting, denoted as V,,vvettor a = (cko, 01,00 00,,, ... Yo, To find the solution, the least squares approximation strategy used minimize the sum of the square errors between the jet value and the height function over the point set P
f(x;: v.) Ja,n (I,, M4), Vi =i.,,
N > ((( Ja.n (xi, 22)) i=1
J argmin
Then the surface fitting problem of Eq. (2) is described finding the least squares solution of homogeneous system of linear equations. Once the n-jet coefficient a is solved, the normal vector at point p the fitted surface is computed by
np = h(a)= = (-a1,0- -0,,1, 1)/ + af, + ag
(4)

---

3.2 Hyper Surface Fitting
The explicit surface fitting method in Sec. 3.1 requires predefined polynomial function with an order n and its performance is susceptible to noise and outliers, as shown in Sec. 4.3 Motivated by the rapidly developed data-driven approaches. which excel at adaptively learning a fitting model that describes the pattern of the provided noisy data [I3 32
36 48) we propose to implicitly learn hyper surfaces in the feature space using Hyper Surface Fitting technique. We expand the 3D point coordinates (I; y. 2) into high dimensional features (G, C, F), and F ~ F(G, C), F E R Then the new nature-bbsed formulation for the polynomial function Jo (I. y) in Eq. 2 given by (see supplementary materials for the derivation)
No.r (,, = Σ k=
Ok 0k-ia Jk- 0 |G: C),
(5)
where [:] means feature fusion operation, such as concatenation. G E R and C' E R are high dimensional features of the 3D point clouds extracted by two different modules, which are introduced in the following two sections. Here, both the parameter T and the coefficient 0 are parameters of sequence of skip-connected Residual Block units (see Fig. 2. Similar with Eq. (3). the bivariate function No.- (G, C) aims to map each feature pair (G,, C.) to their true value F(G,, Ct) E Rº in the feature space
an MLP-based module, which is designed
N NØ = argmin 0,7
Σ -ING,(G..C)-FGG.CC - F(G,,
In contrast to compute point normals formulaically by the n-jet coefficients as in Eq. (4) we recover the normal vectors from the o-dimensional hyper surface using a function H R -R. which is applied to the fitting surface and the surface of fitting sought, i.e. the ground-truth surface
N
N argmin 0,T
Σ
(G, C.)) - ((FFGG,,,)))
Then we get the optimization function about the normals
argmin 8,7
Σ n: n
(8)
where and ñ the output unoriented normal and the ground-truth, respectively. Finally, the normal of the query point p is formulated as the weighted maxpooling of its neighborhoods (see Output Module in Fig. 2
up/ Hell in H(MAX(w; (G, C.)li NJ).
where M. means maxpooling, w) sigmoid (NNg. C))) is the weight, H and V are MLPs. We experimentally find that sin distance [ı x n, is more suitable for measuring the vector difference and guiding the estimated normal to match the ground-truth (see Sec. 4.3). Our method adaptively learns the optimal hyper surface Ng.- from high dimensional features, and more robust than the n-jet fitting, which fits a single type of surface from 3D points with constant order.
3.3 Space Transformation
The previous methods usually use the PointNet-like architectures to learn features from point clouds. Since PointNet is inadequate for encoding local structures, we design a feature extraction module called Space Transformation, which learns from local neighborhood, small and large patch scales. This module provides noise-less global location code G for the hyper surface fitting, and it consists of a sequence of Local Aggregation Layer units and Global Shift Layer units (see Fig. 2}. We repeatedly employ each kind of unit on different levels of feature representation details. The raw point clouds are often noisy and the points have position offsets relative to the noise-free points, which will lead deviation in the learned features. The Local Aggregation Layer delivers smooth filtered relative feature of each point by the cascaded local frame construction and maxpooling operation. Since the relative features only describe the local structures, we design a Global Shift Layer to provide the final features with global position information in the feature space by fusing global features. Specifically, we explore the global information from different scales, as shown in Fig.
5

---

In the Local Aggregation Layer, we first convert the input points to a fixed dimension. Then, we group the local neighborhood features at each point by the 3D spatial dis- tance based kNN search, and refine each grouped feature via chain of Dense Block units [23]. Finally, we compute an order-invariam per per-point feature via maxpooling [42]. In the Dense Block, we use the skip-connection leverage features extracted across different layers. this way, the information of different layers is combined via intra-level connections inside the unit, which realizes the information reuse, thereby improving the learning ability of the network.
Local Aggregation
Glohal Shift
Figure 3: Space Transformation module.
In the Global Shift Layer, we provide the localization of points in the entire patch feature space by fusing global features extracted from multiple neighborhood scales of the query point p. Extracting multi-scale features has become an effective strategy to further boost the normal estimation perfor- M1 [5 The small scale results in more accurate description of the details, while the large scale provides more information about the underlying geometries. summary, our scale-aware Global Shift Layer is formulated as (see the blue part in Fig. 3
mance 10 17 53
G8+1. U, (V,A
G. N.)),
1,...,N+11:
(10)
where U, V are MLP layers, G. is the per-point feature at scale s. MAX(-) means performing feature maxpooling over all neighboring points of p with size N, and No+1 & N..
3.4 Relative Position Encoding
allowing the self-attention mechanism to capture sequence order information of input tokens. We borrow the idea and use it here extract a condition code C' from the local geometry of point cloud data. Contrast the traditional position encoding, which manually craft for language sequence or image grid based on sine and cosine functions, we design a parameterized and learnable encoding scheme which is trained together with the whole model. Given a point p; E {pili= 1,., .M and its neighborhoods {p. 1,= K), our learning- -aased position encoding function @ using relative coordinates is formulated as
Position encoding is important for the transformer architecture 47 18 50
C o(p," Pt: E(p," -p))),
(11)
where p, ps denotes the relative position in local frame, and M N/4 neighboring points of p are used for encoding. The encoding functions & and ø are MLP layers. Experimental results show that our encoding scheme outperforms the traditional position encoding in the normal estimation task.
3.5 Loss Functions
To predict normal vectors that match the ground-truth as close as possible, we minimizes the sin loss between the output unoriented normal n, and the ground-trun normal ñ, the query point p
L ي x npll-
(12)
Meanwhile, we also adopt a weight loss term similar to 44
1
L2= N Σ wm .
(13)
where is the generated point weights in the output module. o; exp(-(p: 2/2) and 5 m0x00.05 0.3 EN (p. fi,))/N), where ps, ñi E RR3. In total, the final loss is given by
L = anLı + a2L2,
(14)
where 01 0.1 and a2 1.0 are weighting factors.
4 Experiments
Datasets and Settings. We first compare our method with other baselines using the same train/test. data of PCPNet shape dataset [17]. To evaluate the generalization ability of each method on the

---

Table I: Normal angle RMSE results the PCPNet and SceneNN dataset. Sorted by the average values on the PCPNet dataset. Lower is better. * means the code is not available or uncompleted.
PCPNet Dataset
SceneNN Dataset
Category
Noise a
Density Stripes
Average
Clean
Noise
Average
None
0.12%
0.6%
1.2%
Gradient
Jet [12
12.35
12.84
18.33
27.68
13.39
13.13
16.29
15.17
15.59
15.38
PCA 21
12.29
12.87
18.38
27.52
13.66
12.81
16.25
15.93
16.32
16.12
PCPNet 17
9.64
11.51
18.27
22.84
11.73
13.46
14.58
20.86
21.40
21.13
Zhou et al."
63
8.67
10.49
17.62
24.14
10.29
10.66
13.62
Nesti-Net 7
7.06
10.24
17.77
22.31
8.64
8.95
12.49
13.01
15.19
14.10
Lenssen ct al. 30
6.72
9.95
17.18
21.96
7.73
7.51
11.84
10.24
13,00
11.62
DeepFit 6
6.51
9.21
16.73
23.12
7.92
7.31
11.80
10.33
13.07
11.70
MTRNet
6.43
9.69
17.08
22.23
8.39
6.89
11.78
Refine-Net 52
5.92
9.04
16.52
22.19
7.70
7.20
11.43
18.09
19.73
18,91
Zhang et al."
49
5.65
9.19
16.78
22.93
6.68
6,29
11.25
9.31
13.11
11.21
Zhou et al."
S4
5.90
9.10
16.50
22.08
6.79
6,40
11.13
AdaFit
55
5.19
9.05
16.45
21.94
6.01
5,90
10.76
8.39
12.85
10.62
Ours
4.17
8.78
16.25
21.61
4.98
4.86
10.11
7.55
12.23
9.89
No Noise
Low Noise
Med Noise
Cleun
PGP
Despitr
Ziäzz ctal.
PCA
6.-
Linson
Bottne-Nr
Noa Net
High Noise
Density: Stripes
Donsity: Gradient
Noise
0.
PGP
Decffit
ol.
PCA
Ratine-Na
PCPNec
Neii-Ne
it
dlucshll&ddggse)))
Threshold deggoo)))))))))))
Tlesshodddeerre)
(a) PCPNet Datasct
(b) SceneNN Dataset
Figure 4:
Normal error AUC results on the PCPNet and SceneNN dataset. X-axis is the angle threshold in degree and Y-axis the percentage good point (PGP) normals under a given threshold.
real-world scene data, we employ the models pretrained on the PCPNet dataset to report results on the indoor SceneNN dataset (22) and the outdoor Semantic3D dataset [19]. For PCPNet dataset, follow the experimental setup in [7 including the train/test set split, noise level, density variations and the data augmentation. We randomly sample 1000 query points for each shape during training. We set the initial point cloud patch size N N -70 and scale set N, = (N, N/2, N/4). The number of kNN points in the Space Transformation is 16 and in the encoding module K = 16. We Adam optimizer with initial learning rate 5 x 10 14-4 and the learning rate is decayed to x0.2 of the latest value after every 200 epochs. The model is trained with batch size of 100 in 900 epochs on an NVIDIA 2080 Ti GPU.
Evaluation Metries. We adopt the widely-used angular Root Mean Squared Error (RMSE) between the predicted normal and the ground-truth to evaluate the normal estimation results. We also use the Area Under the Curve (AUC) to analyze the error distribution of the predicted normal. It is presented by the Percentage of Good Points (PGP) metric, which measures the percentage of point normals with errors below different angle thresholds.
4.1 Results on Synthetic Shape Dataset
Table T reports numerical comparison in terms of RMSE on the PCPNet dataset 17
Fig. M(aa
shows the normal error AUC results. It can be seen that our method outperforms both the traditional
7

---

Ours
it
Refine-Net
DeepFit
Ours
Adal'it
Refine-Net
DeepFit
8.57
11.16
14.34
14.01
6.50
7.46
19.28
9.38
9.44
10.51
21.95
12.75
12.74
16.42
19.05
19.30
8.63
9.53
19.04
11.50
17.46
25.05
21.04
26.72
5.82
6.22
8.70
9.03
5.52
6.11
12.10
7.86
(a) PCPNet Dataset
(b) SceneNN Dataset
Figure 5: Error visualization results on the PCPNet complex shapes and SceneNN dataset. The number under each point cloud is its RMSE. The point color the angular error mapped to heatmap.
methods and the learning- based methods across all noise levels and density variations. The results also show that the performance of all methods degrades substantially as the noise level increases. but the influence of density variations is much smaller than the noise. Fig. ([a) shows the visual comparison on different shapes, where the point clouds are rendered with normal angle error map. We can see that errors mostly occur in regions with complex geometry or high curvature, e.g. edges and corners. Our method shows smallest errors on these regions due to its hyper surface fitting in high dimensional feature space, enabling it to adapt to the different local geometry and disregard irrelevant outliers.
4.2 Results on Indoor and Outdoor Scene Datasets
Table 1 presents numerical comparison with the state- of-the-art results in terms of RMSE on the SceneNN
Raw
Ours
AdaFit
Zhang et al.
DeepFit
Lenssen et al.
dataset [22). Fig. 4(b)
shows the normal angle eтror AUC results. Fig. b)) visualizes the qualitative comparison of errors in different room scenes. The quantitative and qualitative comparison results show that our method outper- forms both the traditional and the learning- methods in all scenes and data categories.
In Fig. 6 we show visual comparison on the Seman- tic3D dataset [19] as the ab- sence of the ground-truth normal. can observe that our method preserves fine details in the complex structures of the building, such as sharp edges, groove between the bricks, pattern and relief on the wall, while other methods behave over-smooth these areas or give wrong normals. The comparison results demonstrate that our method generalizes well to the real-world LiDAR data.
Figure 6: normal vectors are mapped to RGB colors.
Visual comparison on the Semantic3D dataset. The point

---

Table 2: Normal angle RMSE results of ablation studies the PCPNet dataset. The ablation experi- ments include: our new designed modules, i.e. (a) Hyper Surface Fitting, (b) Space Transformation, (c) Relative Position Encoding and (d) Output Module, and hyperparameter, ie. (e) the input point cloud patch size. Please see the text for more details.
Hyper Space
Position Output Trans. Encoding Module
Noise
Density
Ablation
Average
Surface
None
0.12%
0.6%
1.2%
Stripes
Gradient
Jet n
7.55
9.95
16.54
21.99
8.89
7.83
12.12
Jet n
8.52
10.13
16.59
22.12
10.09
9.05
12.75
Jct n =
7.50
9.70
16.68
22.22
9.01
8.07
12.19
Simple
4.44
8.84
16.29
21.76
5.27
5,18
10.30
PointNet-hased
4.84
9.09
16.47
21.98
5.67
5.63
10.61
N' = 256
5.56
9.19
16.76
23.20
6.71
6.21
11.27
N' = 500
5.51
9.24
16.65
22.10
7.11
6.23
11.14
N' = 700
5.80
9.32
16.61 21.86
7.15
6.28
11.17
W/ Traditional
4.43
8.89
16.19
21.64
3,44
5.14
I0.29
w/o Encoding
4.66
8.80
16.32
21.77
5.60
5,37
10.42
w/o Weight
5.04
8.98
16.48
21.80
6.14
5.85
10.71
w/ Loss Leon
4.35
8.90
16.20
21.77
5.34
5.17
10.29
w/ Loss LMSE
10.96
12.68
19.76
24.76
13.26
11.29
15.45
Full N = 600
4.25
8.69
16.20
21.78
5.11
4.92
10.16
(e)
Full N 800
4.30
8.73
16.19
21.53
5.08
5.04
10.15
Final N = 700
4.17
8.78
16.25
21.61
4.98
4.86
10.11
4.3
Ablation Studies
(a) Hyper Surface Fitting. To validate the effectiveness of the proposed hyper surface fitting technique, we compare the performance of different models with the n-jet fitting and our hyper surface fitting. The same backbone network (i.e. space transformation and position encoding) is used in this experiment, but the inputs of the jet fitting based models are 3D points and point-wise weights, which are predicted by the backbone network. We also evaluate model that replaces the residual block in our hyper surface fitting module with a concatenation operation, and we call it "Simple". The results are shown in Table 2 a), we can see that the performance of the models with jet fitting is sensitive to the polynomial order n, while the hyper surface fitting can effectively improve the performance. Moreover, simple 1-order may be enough for the n-jet fitting in the normal estimation task.
(b) Space Transformation Our Space Transformation composed of Local Aggregation Layer units and Global Shift Layer units. In Tbble we conduct feature extraction experiments using two types of the Space Transformation: 1) with PointNet based aggregation layer; 2) without the multi-scale Global Shift Layer and with a fixed scale N'. The results show that our novel components can effectively boost the normal estimation performance.
(c) Encoding Module. In Table c), we provide the results with the traditional position encoding in (47 We also provide the results of model that directly uses 3D point coordinates as the condition code without any encoding processing. The results show that our learnable encoding scheme can improve the performance, and also remove the requirement of selecting specific encoding functions in the traditional one.
(d) Output Module. We use a weighted maxpooling operation in Eq. to recover the query point normal n, from the hyper surface fitting results. For ablation, we estimate the normal without using the weight, i.e., np (MMM((Nee. (G;, Ci
IN...NN N)). Moreover, we also evaluate other two models: 1) trained with an additional weighted neighborhood consistency loss Loon = N N Wi [i x nz/ where the neighboring point normals are predicted by another function n; = H'' - (G,, C,)); 2) trained with the MSE loss LMSE instead the sin loss. The results are shown in Table dd) and we observe that: 1) the weight can improve the results; 2) the further added consistency loss Leon is not helpful for the final results, which means that we only need the query point normal in each point cloud patch and omit the neighboring point normals (also reduce network parameters); 3) the loss LMSF can not guide the model to learn accurate normals.
(e) Input Patch Size. In the previous experiments, we set the input point cloud patch size to 700. In Table e), we evaluate two full models with different patch sizes 600 and N 800. The larger size brings better results than the smaller size under large noise, but takes more time and memory.

---

Point Cloud
Ground-truth
Ours
AdaFit
Zhang et al.
Refine-Net
DoepFit
Figure 7: Visual comparison of the reconstructed surface using the normals estimated by different methods. local enlarged view is provided under each shape.
4.4 Application to Surface Reconstruction
We adopt the Poisson surface reconstruction 2 reconstruct the object surface from the point cloud with point-wise normals estimated by different methods. Fig. [7shows the reconstructed surfaces of different objects. We can see that our estimated normals can facilitate the reconstruction algorithm to produce more accurate and complete surfaces than other baseline methods. In the supplementary materials, we provide more evaluation results and subsequent applications of normal estimation.
5 Conclusion
In this paper, we solved several problems in existing point cloud normal estimation methods. We propose directly regress point normal vectors through a novel hyper surface fitting technique, which implicitly learns hyper surfaces from noisy 3D points in high dimensional feature space, rather than fitting geometric plane or surface using polynomial function with predefined order. Furthermore. we propose two novel modules, i.e. Space Transformation and Relative Position Encoding, to transform 3D point clouds into the feature space from different spatial dimensions. We show that these designs are more effective than their respective counterparts. As result, our HSurf-Net reduces influence of noise and outliers on the fitting process and improves robustness and accuracy of normal estimation. Extensive experiments prove the effectiveness of HSurf-Net, and demonstrate that it produces visually and quantitatively better results than state-of-the-as methods. The future works include exploring more effective point cloud representations, and more efficient local feature extraction for normal estimation.
Acknowledgement
This work was supported by National Key R&D Program of China (2022YFC3800600, 2020YFF0304100; the National Natural Science Foundation of China (62272263, 62072268), and in part by Tsinghua- Kuaishou Institute of Future Media Data.
Tsinghua-Kuaishou
10

---

References
[1] Mare Alexa, Johannes Behr, Daniel Cohen-Or, Shachar Fleishman, David Levin, and Claudio T Silva.
Point set surfaces. In Proceedings Visualization, 2001. V1S pages 21-29 IEEE, 2001.
[2] Pierre Alliez, David Cohen-Steiner, Yiying Tong, and Mathieu Desbrun. Voronoi-hased variational reconstruction of unoriented point sets. In Symposium on Geometry Processing, volume 7. pages 39-48.
2007.
[3] Nina Amenta and Marshall Bern. Surface reconstruction by voronoi filtering. Discrete & Computational Geometry, (144):81-5044 1999.
[4] Samir Aroudj, Patrick Seemann, Fabian Langguth, Stefan Guthe, and Michael Goesele. Visibility-consiscen thin surface reconstruction using multi-scale kernels. ACM Transactions on Graphics, 36(6):1-13 2017.
[5] Haim Avron, Andrei Sharf, Chen Greif, and Daniel Cohen-Or. L.1 -l-ppasee reconstruction of sharp point set
surfaces. ACM Transactions on Graphics, 19 2010.
[6] Yizhak Ben-Shabat and Stephen Gould. DeepFit: 3D surface fitingg via neural network weighted least
squares. In European Conference on Computer Vision, pages 20 20-34. Springer, 2020.
[7] Yizhak Ben-Shabat, Michael Lindenbaum, and Anath Fischer. Nesti-Net: Normal estimation for unstruc- tured 3D point clouds using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 101 12-11220 2019.
[8] James F Blinn. Simulation of wrinkled surfaces. ACM SIGGRAPH Computer Graphics, 12(3):286-292.
1978.
[9] Alexandre Boulch and Renaud Marlet. Fast and robust normal estimation for point clouds with sharp features. In Computer Graphics Forum, volume 31, pages 1765-1774 Wiley Online Library, 2012.
[10] Alexandre Boulch and Renaud Marlet. Deep learning for robust normal estimation in unstructured point clouds. In Computer Graphics Forum, volume 35, pages 281-290. Wiley Online Library, 2016.
[11] Junjie Cao, Hairui Zhu, Yunpeng Bai, Jun Zhou, Jinshan Pan, and Zhixun Su. Latent tangent space representation for normal estimation. IEEE Trunsactions on Industrial Electronics, 69( (911:921-929, 2021. [12] Frédéric Cazals and Marc Pouget. Estimating diffcrential quantitics using polynomial fitting of osculating jets. Computer Aided Geometric Design, 22(2):121-12 2005.
[13] Chao Chen, Yu-Shen Liu, and Zhizhong Han. Latent partition implicit with surface codes for 3D representation. Proceedings of the European Conference on Computer Vision, 2022.
[14] Tamal K Dey and Samrat Goswami. Provable surface reconstruction from noisy samples. Computational
Geometry, 35(1-2)1124-1411 2006.
[15] Henri Gouraud. Continuous shading of curved surfaces. IEEE Transactions on Computers, 10066):623-629
1971.
[16] Gaël Guennehaud and Markus Gross. Algebraic point set surfaces. In ACM SIGGRAPH 2007 papers. 2007.
[17] Paul Guerrero. Yanir Kleiman, Maks Ovsjanikov, and Niloy Mitra. PCPNet: learning local shape properties from raw point clouds. In Computer Graphics Formm volume 37, pages 75-85. Wiley Online Library, 2018.
[18] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. PCT: Point cloud transformer. Computational Visual Media, 172)9187-199, 2021.
[19] Timo Hackel, Savinov, L. Ladicky, Jan D. Wegner, K. Schindler, and M. Pollefeys. SEMANTICSD
A new large-scale point cloud classification benchmark. In ISPRS Annals of the Photogrammetry, Remote
Sensing and Spotial Information Sciences, volume IV-1-W1, pages 91-98, 2017.
[20] Taisuke Hashimoto and Masaki Saito. Normal estimation for accurate 3D mesh reconstruction with point
cloud model incorporating spatial structure. In CVPR Workshops, pages 54-63, 2019.
[21] Hugues Hoppe, Tony DeRose, Tom Duchamp, John McDonald, and Werner Stuctzle Surface reconstruction
from unorganized points. In Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques, pages 71-78, 1992.
[22] Binh-Son Hua, Quang- Hieu Pham, Duc Thanh Nguyen, Khoh Tran, Lap-Fai Yu, and Sai-Kit Yeung. SceneNN: A scene meshes datasct with annotations. In 2016 Fourth International Conference on 3D Vision, pages 92-101. IEEE, 2016.
[23] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1770-47088 2017.
[24] Hui Huang, Dan Li, Hao Zhang, Uri Ascher, and Daniel Cohen-Or. Consolidation of unorganized point clouds for surface reconstruction. ACM Transactions Graphics, 28(5))1-7, 2009.
[25] Hui Huang. Shihao Wu, Minglun Gong. Daniel Cohen-Or, Uri Ascher, and Hao Zhang. Edge-aware point set resampling, ACM Transactions on Graphicx, 32(1):1-12, 2013.
[26] Robert Jacobs, Michael I Jordan, Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991.
[27] Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. In Proceedings
of the fourth Eurographics Symposiun on Geometry Processing, volume 2006.
[28] Carsten Lange and Konrad Polthier. Anisotropic smoothing of point sets. Computer Aided Geometric Design, 22(7):680-692, 2005.
[29] Thibault Lejemble, David Cocurjolly, Loic Barthe, and Nicolas Mellado. Stable and efficient differential estimators on oriented point clouds. In Computer Graphics Forum, volume 40 pages 205-216. Wiley Online Library, 2021.
11

---

[30] Jan Eric Lenssen, Christian Osendorfer, and Jonathan Masci. Deep iterative surface normal estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11247-
11256, 2020.
[31]
David Levin. The approximation power of moving least-squares. Mathematics of Computation, 151224) 1998.
[32] Tianyang Li, Xin Wen, Yu-Shen Liu, Hua Su, and Zhizhong Han. Learning deep implicit functions for 3D shapes with dynamic code clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12840-12850, 2022.
[33] Dening Lu, Xuequan Lu, Yangxing Sun, and Jun Wang. Deep feature-preserving normal estimation for point cloud filtering. Computer-Aided Design, 1 125:102860,, 2020.
[34] Xuequan Lu, Scott Schaefer, Jun Luo, Lizhuang Ma, and Ying He. Low rank matrix approximation for 3D geometry filtering. IEEE Transactions on Visualization and Computer Graphics, 2020.
[35] Xuequan Lu, Shihao Wu. Honghua Chen, Sai-Kit Yeung. Wenzhi Chen, and Matthias Zwicker. GPF: Gmm-inspired ceature-preserving point set filtering. IEEE Transactions on Visualization and Computer Gruphicx. 24(8):2315-2336 2017.
[36] Baorui Ma, Yu-Shen Liu, and Zhizhong Han. Reconstructing surfaces for sparse point clouds with on- surface priors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6315-6325, 2022.
[37] Quentin Mérigot, Maks Ovsjanikov, and Leonidas J Guibas. Voronoi- curvature and feature estimation from point clouds. JEEE Transactions on Visualization and Computer Graphics, 17(6):743-755 2010. [38] Niloy Mitra and An Nguyen. Estimating surface normals in noisy point cloud data. In Proceedings of the
Nineteenth Annual Symposium
Computational Geometry, pages 322-328, 2003.
[39] Mark Pauly, Markus Gross, and Leif P Kobbelt. Efficient simplification of point-sann saint-s surfaces. In
IEEE Visualization, 2002. VIS 2002., pages 033-170. IEEE, 2002.
[40] Bui Tuong Phong. Illumination for computer generated pictures. Communications of the ACM, 18(6):311-
317, 1975.
[41] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pages 652-66 660, 2017.
[42] Charles Ruizhongtai Qi, Yi, Hao Su, and Leonidas Guibas. PointNet++: Deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems, 30:5099-5511 -100, 2017.
[43]
Riccardo Roveri, A Cengiz Öztireli, Ioana Pandele, and Markus Gross, PointProNets: Consolidation of point clouds with convolutional neural networks. In Computer Graphics Forum, volume 37, pages 87-99.
Wiley Online Library, 2018.
[44]
Michael D Spivak. A comprehensive introduction to differential geometry. Publish perish, 1970.
[45]
Gilbert W Stewart. On the early history of the singular value decomposition. SIAM Review, (5(4):551-566,
1993.
[46] Yujing Sun, Scott Schaefer, and Wenping Wang. Denoising point sets via 10 minimization. Computer
Aided Geometric Design, 35:2-15, 2015.
[47]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems,
30, 2017.
[48] Xin Wen, Junsheng Zhou, Yu-Shen Liu, Hua Su, Zhen Dong, and Zhizhong Han. 3D shape reconstruction from 2D images with disentangled attribute flow. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3803-3813, 2022.
[49] Jie Zhang, Jun-Jie Cao, Hai-Rui Zhu, Dong-Ming Yan, and Xiu-Ping Liu. Geometry guided deep surface normal estimation. Computer-Aided Design. 142: 103119, 2022.
[50] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 16259-16268,, 2021. [51] Haoran Zhou, Honghua Chen, Yidan Feng, Qiong Wang. Jing Qin, Haoran Xie. Fu Lee Wang. Mingqiang Wei, and Jun Wang. Geometry and lcarning co-supported normal estimation for unstructured point cloud, In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 00238-1327,, 2020.
[52] Haoran Zhou, Honghua Chen, Yingkui Zhang, Mingqiang Wei, Haoran Xie, Jun Wang. Tong Lu, Jing Qin, and Xiao-Ping Zhang. Refine-Net: Normal refinement neural network for noisy point clouds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[53] Jun 7hou, Hua Huang, Bin Liu, and Xiuping Liu. Normal estimation for 3D point clouds via local plane constraint and multi-scale selection. Computer-Aided Design, 129:102916, 2020.
[54] Jun Zhou, Wei Jin, Mingiie Wang, Xiuping Liu, Zhiyang Li. and Zhaobin Liu. Improvement of normal estimation for point clouds via simplifying surface fitting. arXiv preprint arXiv:2 10369, 2021. [55] Runsong Zhu, Yuan Liu, Zhen Dong, Yuan Wang, Tengping Jiang, Wenping Wang, and Bisheng Yang. AdaFit: Rethinking learning-based normal estimation on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6118 6127, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In the supplemental materials.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In the supplemental materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] It would be too computationally expensive.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes)
(b) Did you mention the license of the assets? [No] We were unable to find the license for the dataset we used.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "zYiC2E_Ncws", "reviewer": "Reviewer_KzzY", "summary": "The authors propose a novel normal estimation method called HSurf-Net. The method works by implicitly learning hyper surfaces in a high dimensional feature space. The method includes a novel space transformation module consisting of a sequence of local aggregation layers and global shift layers, to reliably build the feature space. Experimental results show that the HSurf-Net can accurately predict normals from point clouds with noise and density variations, and achieves state-of-the-art performance on multiple datasets.", "questions": "- There are a lot of different notations (especially hyper-parameters) in this paper. Organizing them into a table in the appendix may help readers to follow the paper more easily.\n- The idea of hierarchical point set abstraction (Fig.3) is actually proposed by PointNet++. The authors might want to cite this paper.\n- Include a real-world example (e.g. a lidar scan) could further strengthen the paper.\n- According to the theory proposed in this paper. The points are lifted to noise free features before fitting hyper surfaces. Could we utilize this feature to perform other point processing tasks (maybe jointly) e.g. denoising / super-resolution? Any discussions on this would be appreciated.", "limitations": "Yes, in the supplementary material.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well-written and easy to follow. The method is technically sound. The authors have done abundant experiments covering a broad range of baselines and datasets, making the results convincing.\n- The idea of transforming points into a higher dimensional feature space and fitting hyper surfaces is interesting and novel. The authors have designed the network accordingly to achieve this goal and performed ablation studies to validate each design choice.\n- The method has shown impressive robustness towards noisy inputs. The HSurf-Net outperforms all other baselines under all settings.\n\nWeaknesses\nThis paper is well-presented and proposes a novel model for normal estimation with state-of-the-art performance. I do not see any major issue in its current form. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LIuBDxA9Ek", "reviewer": "Reviewer_o6yD", "summary": "- This paper introduces an approach to surface normal estimation from the 3D point clouds. Instead of applying function fitting to the input data that may undergo overfitting or underfitting, the approach introduces hyper surface fitting to learn hyper surfaces in the high dimensional feature space implicitly. The approach learns hyper surfaces from the features and directly predicts normal vectors. The approach (HSurf-Net) is validated with the synthetic and real-world indoor and outdoor datasets.", "questions": "1. Figure 5 is hard to distinguish the proposed approach. Is there any way to visualize the errors more effectively? How about utilizing log-scale errors or changing the color map? Similarly, Figure 4 could be improved as well. Because many plots are overlapped, consider adjusting the min and max of PGP to show the other plots better. Consider using other line styles to distinguish the lines better.  \n2. Another suggestion is to apply the estimated normal to the various applications. For instance, making a mesh from a raw point cloud remains a challenging problem. How about showing that the noisy raw point clouds can be successfully reconstructed as a mesh using the proposed approach? Maybe using the Poisson reconstruction using the various estimation of surface normal?\n3. Please answer the questions in the paper weakness section.\n", "limitations": "- The paper does not address the limitation of the proposed approach. The concern about the network overparameterization and the issue about point cloud densities should be clarified and stated in the limitation section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper revisits the normal estimation problem and considers the problem as the hyper-surface fitting problem. Since the surface fitting happens on the latent feature domain, such implicit fitting is rather robust to the explicit approaches that are easily affected by additive noise. The reviewer is not fully tracking the literature on this field, but it seems novel and interesting.\n2. The paper comprehensively summarizes the related work in the normal estimation field. The paper is self-contained, so the readers readily follow the problem and the recent advances in this field.\n3. The paper is straightforward to understand, and the approach shows compelling results on the PCPNet, SceneNN, and Semantic3D datasets. The visual quality of the estimated normal is reasonable.\n4. The paper explains the proposed idea in detail. The supportive figures (such as Figures 2 and 3) help to understand the approach better.\n\n**Weakness**\n1. It is unclear that the proposed approach is completely free from ‘underfitting and overfitting'. The results may depend on the ‘c’ dimension of the hyper surface coefficients (Sec. 3.2). Even if the relevant features and network output would fit for the ‘c’, there is a danger that network capacity would be overparameterized. Can the authors mention this? This is an important issue because the main argument of the proposed approach is expressed in this manner. For instance, lines 7-9 “fitting surfaces explicitly from arw point clouds suffers from overfitting or underfitting issues caused by inappropriate polynomial orders and outliers… To address these issues, we introduce hyper surface fitting to implicitly learn hyper-surfaces…”\n2. The relative position encoding module directly embeds the relative position in a local frame (p^j_j-p_i). However, such embedding would work for the specific metric scale. How would the approach handle the point clouds with non-metric scale point clouds? What I mean is that the scale set of the patch size {N, N/2, N/4} would be invariant regardless of the scale of the point cloud, but (p^j_j-p_i) would be directly affected by the scale of the point cloud. In other words, if there are two point clouds {X} and {X/10}, can the network would produce the same output?\n3. It is interesting to see that Eq (9) induces the normal vector. However, the equation itself does not guarantee ||n||=1 because it is the network output of the max-pooled feature vectors. Does any additional post-processing is applied to get the normalized n? Please clarify.\n4. One critical concern is whether the approach would handle uneven point clouds. The paper mentions the ‘patch’ over the paper, and mention that ‘patch size’ as the number of the points (Line224 and so on). However, such a definition of ‘patches’ is readily affected by the density of the point cloud. For instance, in the KITTI dataset, in the coarse region, if we set N=700, it will introduce severely biased point cloud samples. The all experiments are conducted with the evenly sampled point clouds, where the density of the point cloud is fixed over the dataset. Please state how the proposed approach would handle density changes or uneven point clouds.\n5. In the experiment section, the paper states the proposed approach outperforms the previous approaches, but it would be better if the paper mentioned the comparison between the second-best approach. For instance, AdaFit[45] is quite similar to the proposed approach. Please provide some discussion or analysis of the suggested approach.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B2jf6KM7boa", "reviewer": "Reviewer_Md6E", "summary": "This paper proposes a new method for the task of normal estimation for point clouds using hyper-surface fitting. The model consists of relative positional encoding, and a space transformation module (MLP) to map 3D point clouds into feature space where the hyper-surfaces are fitted. The method is evaluated on three different datasets, including PCPNet, ScenNN, and Semantic 3D and shows state-of-the-art performance in RMSE and PGP metrics. ", "questions": "There are several aspects that I am not clear on: \n1. The space transformation module is unclear to me and I found Figure 3 confusing. How is it different from a PointNet++ idea? While clearly, the architecture is different the concept of multi-scale and grouping was first proposed there, so how is this different? \n2. Novelty: To my understanding, this method could be interpreted as a newer version of PCPNet with PointNet++ as backbone architecture and Positional encoding as input. Each has already been proposed in previous works and showed its effectiveness and the question is how is combining these building blocks novel? \n3. Insight: The main paper lacks insight into what the network is actually learning. There is some of it in the supplemental but not sure that it is enough. It would be interesting to visualize each point's contribution to the hypersurface fitting as a heatmap. \n4. Will the code be released ? \n", "limitations": "The limitations are not clearly presented. Clearly, one limitation is robustness to noise. What are other limitations? When does it fail in the clean point cloud cases? Additionally, no potential societal impact was mentioned (since it is a normal estimation method, I am not sure there are any but even if there aren't, it should at least be mentioned). ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is mostly well written and easy to follow. The method is useful as shown by the results on multiple datasets. Different elements of the model are well analysed by an ablation study and show the contribution of each of the components. Figures 1 and 2 help the reader follow along and clarify the overall framework. The supplemental material also contains interesting results. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8CIYA-yc_1Q", "reviewer": "Reviewer_293b", "summary": "The authors propose a method to estimate normals in point clouds based on a novel patch-based network architecture that integrates global patch information and local neighborhood information inside the patch in multiple steps, making heavy use of densenets and skip connections. A new cross-product-based loss is used to train the network.\n\nI would consider the novel architecture, as well as the novel loss (which seems to be very relevant to achieve good performance according to the ablation) to be the main contributions.", "questions": "The authors should clearly state in the rebuttal if they plan to change the motivation of their method as described above, or alternatively if I missed something that justifies the current motivation given by the authors.", "limitations": "The authors have sufficiently addressed limitations and societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Normal estimation should be of great interest to the community.\n- The architecture is reasonably novel. It roughly follows a PointNet-like setup in most components, but the component combination into the full architecture seems novel.\n- The evaluation shows clear improvements over the state-of-the-art on multiple datasets, including real-world data.\n- There is a thorough ablation of the design choices (even if their theoretical motivation provided by the authors is misleading, this provides some empirical motivation).\n- The authors include source code and promise to provide their exact data.\n\nWeaknesses:\n- The motivation and presentation of the method is misleading.\n- The exposition is missing some details of the method.\n- Given that the current motivation is misleading, this leaves the design choices for the architecture poorly motivated.\n\nOverall, due to the promising that show a clear improvement over the state-of-the-art on several datasets, I am leaning towards the positive side. However, the misleading exposition and missing details lower the quality of the paper significantly and should be corrected before the paper can be accepted. I believe that this can be done in a minor revision.\n\nDetails:\n- The motivation and presentation of the method is misleading. In the introduction, Section 3.1 and Section 3.2, the approach is carefully motivated as surface fitting in a feature space. However it seems like surface fitting is not done explicitly, and instead a generic network is used that may or may not learn to do something similar as the polynomial surface fitting described in Eq. 5. The surface mathcal{N} is never explicitly constructed or encoded in any specific way in the network structure. Considering this I would clearly count the proposed method as regression-based method in the author's terminology, not as a hybrid or fitting-based method. The motivation of the approach as surface fitting seems to me like carefully motivating a method as specifically using a Sobel Filter to detect edges, and then training a generic CNN to obtain the output. It seems quite misleading and should be changed.\n\n- The exposition is missing some details and could be improved.\n  - In Section 3.1, the use of two Taylor expansions seems unnecessarily complicated. Also, since both are defined with the same order it does not become clear that the fitted surface can usually only be an approximation of the true surface. It might be clearer to define the ground truth surface with a generic function f(x,y) instead of the truncated Taylor expansion J_beta,n, and then fit J_alpha,n to this generic function.\n  - The introduction should clearly mention that the goal is to find *unoriented* normals, i.e. without giving information which side of the surface is inside or outside.\n  - In Eq. 5, the dimensionality of the feature space of F is not defined (the output space of mathcal{N}_theta,tau and mathcal{F})\n  - The text below Eq. 5 mentions basis vectors that for G and C that are introduced in later sections, but I could not find mention of them in the rest of the text.\n  - Eq. 6 is missing a vector norm for the difference between feature vectors.\n  - Also, in Eq. 6, would optimizing over tau not always result in the highest possible tau being used? Since higher-order should always be able to achieve a lower error. (This may not be very relevant because no optimization is done in practice - see concerns about the motivation of the method.)\n  - Section 3.2 should clarify how weights w_i are computed.\n  - Losses should be discussed in the main paper, since they are non-standard and seem to be an essential part of the contribution. Shortening or changing Sections 3.1 and 3.2 could make space.\n  - For Eq. 9, some motivation should be given for the design choice of using the max over all features, since this breaks the analogy to 3D surface fitting, where it would not make sense to take the max over all z values of the fitted surface.\n  - In Section 3.3, it should be clarified if the kNN search for neighbors is performed in the input 3D space or the feature space. Also in Section 3.3, it should be clarified if the dense blocks are applied per-point or over the concatenated features of all grouped points in a neighborhood. If it is per-point, the local aggregation layer could be described more succinctly as a PointNet over the local neighborhood, with dense blocks instead of MLPs.\n  - Section 3.3 is missing a description how G_s,i is converted to G_i. Is the last (coarsest) scale used? This should be clarified.\n  - Section 3.3 and 3.4 are missing descriptions how the coarser subset of M points is chosen from the input N points. Farthest point sampling like in PointNet++? A random subset? This should be clarified.\n  - In Section 3.4, the position encoding function phi is not defined. Is it an MLP as well? This should be clarified.\n  - Section 3.4 is missing a description of how C^j_i is converted to C_i. Maxpooling? This should be clarified.\n  - Section 4 should briefly mention how many point clouds were used from each dataset and the average point cloud size.\n  - Section 4 should clarify if all baselines were trained on the same dataset.\n\n- In the ablation, how was the MSE loss applied to the unoriented normals? This should be clarified (a naive application to output normals would not work since the normals are not oriented).\n\n- The following papers could be added to the related work:\n  - Stable and efficient differential estimators on oriented point clouds, Lejemble et al., SGP 2021\n  - PCT: Point Cloud Transformer, Guo et al., CVM 2021\n  - Point Transformer, Zhao et al., CVPR 2021 (they don't demonstrate normal estimation, but their architecture is for general point cloud processing)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept