Title: Inferring Neural Signed Distance Functions by Overfitting on Single Noisy Point Clouds through Finetuning Data-Driven based Priors
Venue: NeurIPS 2024
Accepted: yes
Keywords: point cloud

SDF

3D reconstruction
Abstract: It is important to estimate an accurate signed distance function (SDF) from a point cloud in many computer vision applications. The latest methods learn neural SDFs using either a data-driven based or an overfitting-based strategy. However, these two kinds of methods are with either poor generalization or slow convergence, which limits their capability under challenging scenarios like highly noisy point clouds. To resolve this issue, we propose a method to prompt pros of both data-driven based and overfitting-based methods for better generalization, faster inference, and higher accuracy in learning neural SDFs. We introduce a novel statistical reasoning algorithm in local regions which is able to finetune data-driven based priors without signed distance supervision, clean point cloud, or point normals. This helps our method start with a good initialization, and converge to a minimum in a much faster way. Our numerical and visual comparisons with the stat-of-the-art methods show our superiority over these methods in surface reconstruction and point cloud denoising on widely used shape and scene benchmarks. The code is available at https://github.com/chenchao15/LocalN2NM.

Inferring Neural Signed Distance Functions by Overfitting on Single Noisy Point Clouds through Finetuning Data-Driven based Priors

Abstract

It is important to estimate an accurate signed distance function (SDF) from a point cloud in many computer vision applications. The latest methods learn neural SDFs using either a data-driven based or an overfitting-based strategy. However, these two kinds of methods are with either poor generalization or slow convergence, which limits their capability under challenging scenarios like highly noisy point clouds. To resolve this issue, we propose a method to promote pros of both data-driven based and overfitting-based methods for better generalization, faster inference, and higher accuracy in learning neural SDFs. We introduce a novel statistical reasoning algorithm in local regions which is able to finetune data-driven based priors without signed distance supervision, clean point cloud, or point normals. This helps our method start with a good initialization, and converge to a minimum in a much faster way. Our numerical and visual comparisons with the state-of-the-art methods show our superiority over these methods in surface reconstruction and point cloud denoising on widely used shape and scene benchmarks. The code is available at https://github.com/chenchao15/LocalN2NM .

1 Introduction

It is an important task to estimate an implicit function from a point cloud in computer graphics, computer vision, and robotics. An implicit function, such as a signed distance function (SDF), describes a continuous 3D distance field to indicate distances to the nearest surfaces at arbitrary locations. Since point clouds are easy to obtain, they are widely used as an information source to estimate SDFs, particularly without using normals that are not available for most scenarios. The challenge for SDF estimation mainly comes from the difficulty of bridging the gap between the discreteness of point clouds and the continuity of implicit functions.

Recent methods [62, 64, 29, 14, 95, 80, 58, 74] overcome this challenge using either a data-driven based or an overfitting-based strategy. To map a point cloud to a signed distance field, the data-driven based methods [60, 27, 36, 45, 81, 79, 22, 42, 92, 83] rely on a prior learned with signed distance supervision from a large-scale dataset, while the overfitting-based methods [28, 1, 102, 2, 99, 4, 21, 50, 18, 88] do not need signed distance supervision and just use the point cloud to infer a signed distance field. However, both of the two kinds of methods have pros and cons. The data-driven based methods can do inference fast but suffers from the need of large-scale training samples and poor

generalization to instances that are unseen during training. Although the overfitting-based methods have a better generalization ability and do not need the large-scale signed distance supervision, they usually require a much longer time to converge during inference. The cons of these two kinds of methods dramatically limit the performance of learning neural SDFs under challenging scenarios like highly noisy point clouds. Therefore, beyond pursuing higher accuracy of SDFs, how to balance the generalization ability and the convergence efficiency is also a significant issue.

To resolve this issue, we propose to learn an SDF from a single point cloud by finetuning data-driven based priors. Our key idea is to promote the advantages of both the data-driven based and the overfitting-based strategy to pursue better generalization, faster inference, and higher accuracy. Our method overfits a neural network on a single point cloud to estimate an SDF with a novel loss without using signed distance supervision, clean point, or point normals, where the neural network was pretrained as a data-driven based prior from large-scale signed distance supervision. With finetuning priors, our method can generalize better on unseen instances than the data-driven based methods, and also converge much more accurate SDFs in a much faster way than the overfitting-based methods. Moreover, our novel loss for finetuning the data-driven based prior can conduct a statistical reasoning in a local region which can recover more accurate and sharper underlying surface from noisy points. We report numerical and visual comparisons with the state-of-the-art methods and show our superiority over these methods in surface reconstruction and point cloud denoising on widely used shape and scene benchmarks. Our contributions are summarized below,

We introduce a method which is capable of funetuning a data-driven based prior by minimizing an overfitting-based loss without signed distance supervision, leading to neural SDFs with better generalization, faster inference, and higher accuracy.

The proposed overfitting-based loss can conduct a novel statistical reasoning in local regions, which improves the accuracy of neural SDFs inferred from noisy point clouds.

Our method produces the state-of-the-art results in surface reconstruction and point cloud denoising on the widely used benchmarks.

2 Related Works

Learning implicit functions has achieved promising performance in various tasks [62, 64, 29, 14, 95, 80, 58, 74, 30, 31, 33]. We can learn neural implicit representations from different supervision including 3D supervision [61, 69, 59, 17], multi-view images [78, 44, 38, 101, 46, 94, 63, 41, 98, 97, 25, 86, 100, 89, 84, 85], and point clouds [92, 43, 60, 27]. We briefly review the existing methods related to point clouds below.

2.1 Data-driven based Methods

In 3D supervision, many techniques utilize a data-driven approach to learning priors, and then apply these learned priors to infer implicit models for unseen point clouds. Some strategies focus on acquiring global priors [60, 27, 36, 45, 81, 79, 22, 42] at the shape level, whereas others aim to boost the generalization of these priors by learning local priors [92, 83, 11, 37, 6, 51] at the component or patch level. These learned priors facilitate the marching cubes algorithm [47] to reconstruct surfaces from implicit fields. The effectiveness of these methods often rely on extensive datasets, but they may not generalize well when facing with unseen point clouds that significantly deviate in geometry from training samples.

2.2 Overfitting-based Methods

In an effort to enhance generalization, some methods concentrate on precisely fitting neural networks to single point clouds. These methods incorporate innovative constraints [28, 1, 102, 2, 99, 4, 21], utilize gradients [50, 18, 88], employ differentiable Poisson solvers [70], or apply specially tailored priors [51, 54] to learn either signed [50, 28, 1, 102, 2, 15, 56, 13] or unsigned distance functions [18, 104, 103]. Despite achieving significant advances, these approaches typically require clean point clouds to accurately determine distance or occupancy fields around the point clouds.

Neural

Implicit

Function

f

…

Parameter

Initialization

D

Local

Denoising

…

…

Neural

Implicit

Function

′

′

Query

c

Clean Dataset

Average

Code

Querys

Local

Noisy Point Cloud

M

KNN

KNN

Data-Driven

based Prior

Training

Inference

q

Initialization

Query Points

…

2.3 Learning from Noisy Point Clouds

The key to accurately reconstructing surfaces on noisy point clouds is to minimize the effect of noise in inferring implicit functions. PointCleanNet [73] was developed to filter out noise from point clouds through a data-driven approach. GPDNet [72] incorporated graph convolution based on dynamically generated neighborhood graphs to enhance noise reduction. Some other methods leveraged point cloud convolution [6], alternating latent topology [90, 57], semi-supervised strategy [106, 19], dual and integrated latent [76], or neural kernel field [91, 35] to reduce noise from point clouds. On the unsupervised front, TotalDenoising [10] adopts principles similar to Noise2Noise [40], utilizing a spatial prior suitable for unordered point clouds. DiGS [3] employs a soft constraint for unoriented point clouds. Noise2NoiseMapping [52] leverage statistical reasoning among multiple noisy point clouds with specially designed losses. Some methods using downsample-upsample frameworks [48], gradient fields [49, 9, 16, 68, 65], convolution-free intrinsic occupancy network [67], intra-shape regularization [66], eikonal equation [96, 23], neural Galerkin [34] and neural splines [93] have been implemented to further diminish noise in point clouds. Our method falls in this category, but we aim to promote the advantages of both the data-driven based and the overfitting-based strategy to pursue better generalization, faster inference, and higher accuracy.

3 Method

Overview. We aim to infer a neural SDF f from a single point cloud with noises M . Our method includes two stages shown in Fig. 1, one is to learn a prior f ′ in a data-driven manner, the other is to infer a neural SDF f on unseen noisy point cloud M . At the first stage, we learn a prior by training a neural SDF using ground truth signed distances of clean meshes indicated by embeddings c ′ j . At the second stage, we finetune the learned prior f ′ to infer a neural SDF f of M using our proposed local noise to noise mapping, where the embedding c indicating M is also learned. We can use the marching cubes algorithm [47] to extract the zero-level set of f as the mesh surface of M .

Neural Signed Distance Function. We leverage an SDF f to represent the geometry of a shape. An SDF f is an implicit function that can predict a signed distance s for an arbitrary location q , i.e., s = f ( q ) . The latest methods usually train a neural network to approximate an SDF from signed distance supervision or infer an SDF from 3D point clouds or multi-view images. A level set is an iso-surface formed by the points with the same signed distance value. For instance, zero-level set is a special level set, which is formed by points with a signed distance of 0. On the zero-level set, the gradient ∇ f ( q ) of the SDF f at an arbitrary location q is also the surface normal at q .

Data-driven Based Prior. As shown in Fig. 1, we employ an auto-decoder similar to DeepSDF [69] for learning a prior f ′ in a data-driven manner and inferring a neural SDF f for single point clouds with noises, respectively. We employ a data-driven strategy to learn a prior f ′ from clean meshes first. Specifically, we learn f ′ with an embedding c ′ j as a condition of queries. For each shape, we sample queries q around a shape represented by c ′ j , and establish the signed distance supervision by recording the signed distance s to the ground truth mesh. Thus, we learn the prior f ′ by minimizing the prediction errors to the ground truth signed distances,

where c ′ i is a learnable condition for the i -th training shape, q j is the j -th query that is randomly sampled around the i -th shape, and s j i is the ground truth signed distance. We also add a regularization term on the learned embeddings c ′ i , and α is the balance weight.

Signed Distance Inference. With the learned prior f ′ , we infer a neural SDF f for a single point cloud with noises M . We do not require ground truth signed distances, clean point clouds, or even point normal during the inference of f . Specifically, we infer f by finetuning parameters of f ′ with a learnable embedding c indicating the single point cloud with noises. The finetuning relies on a novel statistical reasoning algorithm on local regions.

The advantage of our method lies in the capability of conducting the statistical reasoning in local regions. Comparing to the global reasoning method [52], our method is able to not only infer more accurate geometry but also significantly improve the efficiency. Our method starts from randomly sampling a local region m n on the shape M . We randomly select one point on M as the center of m n , and set up its K nearest noisy points as a local region m n . Then, we randomly sample U queries { ¯ q u } U u =1 around m n , and also randomly select U noisy points { p v } U v =1 out of m n for statistically reasoning the surface in each iteration.

Our key idea of inferring a neural SDF f is to estimate a mean zero-level set that is consistent to all points in the local region m n . To this end, we use the U sampled queries { ¯ q u } to represent the zero-level set in this area using f , and minimize the distances of the U noisy points { p v } to the zero-level set in each iteration. Statistically, the expectation of the zero-level set should have the minimum distance to all the noisy point splitting in region m n .

Specifically, we first project the U sampled queries { ¯ q u } onto the zero-level set of f using a differentiable pulling operation [50]. For each query ¯ q u , its projection on the zero-level set is,

where ¯ q ′ u is the projection of ¯ q u on the zero-level set, s = f (¯ q u , c ) , ∇ f (¯ q u , c ) is the gradient of f at the location ¯ q u , and c is the learnable embedding that represents the noisy point cloud M .

With the pulling operation, we can use projections { ¯ q ′ u } of queries { ¯ q u } to approximate the zero-level set in region m n . With a coarse zero-level set estimation, we expect this zero-level set can be consistent to various subsets of noises { p v } sampled from m n . Thus, we minimize the errors between the { ¯ q ′ u } U u =1 and a subset of points { p v } U v =1 on area m n in each optimization iteration,

where we learn f through finetuning the prior f ′ and learning the embedding c representing the noisy point cloud M . The expectation is over the local regions m n that randomly sampled from the noisy point cloud M , and the subset patch p v randomly sampled from each m n . We follow the method [52] to use the EMD to evaluate the distance between the two sets of points, which leads the neural SDF f to converge on the specific noisy point cloud M .

Initialization. The network architecture of f is the same to the one of prior f ′ . We learn f with the parameters of f ′ as the initialization, representing the prior that we learned. For the embedding c that represents M , we initialize c as the center of the embedding space learned by the prior f ′ in Eq. 1, i.e., c = 1 /I ∑ I i =1 c ′ i . This initialization is important for the accuracy and efficiency of learning f for single noisy point cloud M . This finetuning of parameters of f ′ also shows advantages over the auto-decoding [69] in terms of generalization and efficiency. We will justify these advantages in our experiments.

Implementation Details. We randomly select one point from noisy point cloud M as a center, and select its K = 1000 nearest points to form a local region m n . We also randomly sample U = 1000 queries around the K noisy points for statistically reasoning. Specifically, we adopt a method

introduced by NeuralPull [50] to sample queries around each one of the K noisy points. We use a Gaussian distribution centered at each point and set the standard deviation as the distance to the 51th nearest neighbor in the point cloud. We run the marching cubes for surface reconstruction at a resolution of 256 for shapes, and 512 for large-scale scenes.

The length of the embedding c or c ′ is set to 256 . We use Adam optimizer for learning a neural implicit network, which is an auto-decoder similar to DeepSDF [69]. For training, we use an initial embedding learning rate of 0.0005 for updating embeddings and an auto-decoder learning rate of 0 . 001 for optimizing the prior network. Both learning rates are decreased by 0.5 for every 500 epochs. We train the prior network f ′ for 2000 epochs. For inference, we finetune the network f ′ for each noisy point cloud in 4000 iterations with a learning rate of 0 . 0001 .

4 Experiments and Analysis

We compare our method with the latest methods in terms of numerical and visual results on synthetic point clouds and real scans in surface reconstruction.

Datasets and Metric. We use eight datasets including shapes and scenes in the evaluations. For shapes, we conduct experiments under five datasets including ShapeNet [12], ABC [22], FAMOUS [22], Surface Reconstruction Benchmark (SRB) [92] and D-FAUST [5]. For scenes, we conduct experiments under three real scan datasets including 3D Scene [105], KITTI [26], Paris-rueMadame [75], and nuScenes [8]. We leverage L1 Chamfer Distance ( CD L 1 ), L2 Chamfer Distance ( CD L 2 ) to evaluate the error between the reconstructed surface and ground truth. We also use Normal Consistency (NC) [59] and F-Score [82] with a threshold of 1% to evaluate the normal accuracy of the reconstructed surface. In the ablation study, we also report time consumption to highlight the superiority of our data-driven based prior. For KITTI and Paris-rue-Madame datasets, due to their lack of ground truth meshes, we only report visual comparisons.

4.1 Surface Reconstruction for Shapes

Evaluation on ShapeNet.

We first report our results on shapes from ShapeNet. We report evaluations by comparing our method with the latest prior-based and overfitting-based methods in Tab 1. For prior-based methods, we compare our method with PSG [24], R2N2 [20], COcc [71], OCNN [87], IMLS [45], POCO [7], and ALTO [90]. All of these methods are pretrained to learn priors using shapes with noises in training set of ShapeNet. We also follow these meth-

POCO

N2NM

Ours

GT

ods to use the same set of training shapes to learn our prior. For overfitting-based methods, we compare our method with PSR [39], SAP [70], and N2NM [52]. These methods did not need to learn a prior, and have the ability of inferring neural implicit functions on each shape in the testing set. We also follow these methods and report our results by finetuning our prior through overfitting on each testing shape. All the shapes for testing are corrupted with noises with a variance of 0 . 005 .

The comparisons in Tab. 1 indicate that our method can infer much more accurate neural implicit functions than the prior-based methods. The improvement comes from the ability of conducting test time optimization with the learned prior and inferring signed distances using the local noise to noise mapping. Moreover, our

Metrics

SAP [70]

N2NM [52]

Ours

Time

14 min

46 min

5 min

local statistical reasoning not only achieves better ability of recovering geometry from noisy points than overfitting-based methods but also significantly reduces the time complexity during the test time overfitting procedure with our prior. Different from prior-based methods, our ability of conducting testtime optimization with our local statistical reasoning loss can significantly improve the generalization ability on unseen shapes. Tab. 2 shows that our method can infer neural implicit functions on single shapes much faster than the overfitting-based methods. We also demonstrate our advantages in visual comparisons in Fig. 2.

Evaluation on ABC. We also report our evaluations on ABC dataset in Tab. 3. We learn priors from shapes in training set, and finetune this prior for each single shape in the testing set. The numerical comparisons are conducted on the testing set of ABC dataset released by P2S [22]. It includes two versions with different

noise levels. Similarly, we also report comparisons with prior-based methods and overfitting-based methods. With our local noise to noise mapping, we achieve the best performance over all baselines. Compared to prior-based methods, such as P2S [22], COcc [71], and POCO [7], our loss can infer more accurate geometry during the test time overfitting procedure. Also, the ability of finetuning the prior can also provide a coarse estimation and a good start for inferring neural implicit from single noisy points. Besides the accuracy, we also observe improvements on efficiency. Fig. 3 demonstrates the improvements over the baselines in terms of surface completeness and edge sharpness.

Evaluation on SRB. We report previous experiments using man-made objects in ShapeNet and ABC dataset, We also report our results on real scans on SRB dataset [92]. Since there is no training samples on SRB, we use the prior learned from the ShapeNet as the prior for real scans. Although the shapes in ShapeNet are not similar to shapes in SRB, we found the prior can also work well with the scans on SRB. Different from the man-made objects, real scans have unknown noises. We report the

evaluations with the prior-based and overfitting-based methods in Tab. 4 and Fig. 4. The comparisons show that our method achieves the best performance in implicit surface reconstruction. Under the same experimental settings, our method can infer more accurate geometry details with our local noise to noise mapping.

Evaluation on FAMOUS. We report evaluations on more complex shapes on FAMOUS dataset. Similar to SRB, we also use the prior learned from ShapeNet. We evaluate the performance on two kinds of noises in Tab. 5. We can see that our method can recover more geometry details and achieve higher accuracy and smoother surfaces. We also report visual comparisons in Fig. 5, which also highlights our improvements in

Input

IMLS

LPI

GridPull

Ours

GT

terms of accuracy, smoothness, completeness, and recovered sharp edges.

Evaluation on D-FAUST. Finally, we report our results on non-rigid shapes, i.e., humans. Different from rigid shapes in the previous experiments, humans are with more complex poses. We learn a prior from the training set, and finetuning the prior on unseen humans with different poses. We mainly compare our method with overfittingbased methods in Tab. 6. We can see that our method achieves the best performance in CD, F-Score, and comparable performance to N2NM [52] but with faster inference speed. We further show the visual comparison in Fig. 6. We can see that our method can recover more accurate geometry and poses.

Input

IGR

Point2Mesh

SAP

Ours

GT

4.2 Surface Reconstruction for Scenes

Since we have a limited number of scenes for training, we use the prior learned from ShapeNet as the pretrained prior in our experiments on scenes. Specifically, we conduct experiments on four different scene datasets: 3D Scene [105], KITTI [26], Paris-rue-Madame [75] and nuScenes [8], where the results on nuScenes are reported in the appendix.

Evaluation on 3D Scene. We further evaluate our method in surface reconstruction for scenes in 3D Scene [105]. We follow previous methods LIG [37] to randomly sample 1000 points per m 2 . We compare our method with the latest methods including COcc [71] and LIG [37], DeepLS [11], NeuralPull (NP) [50] and Noise2NoiseMapping (N2NM) [52]. For prior-based methods COcc [71] and LIG [37], we leverage their released pretrained models to produce the results, and we also provide them with the ground truth point normals. For overfitting-based methods DeepLS [11], NP [50] and N2NM [52], we overfit them to produce results with the same noisy point clouds. We follow LIG [37] to report CD L 1 , CD L 2 and NC for evaluation. We report the comparisons in Tab. 7. The results demonstrate that our method outperforms both kinds of methods with learned priors such as LIG [37] and overfitting-based N2NM [52]. The visual comparisons in Fig. 7 show that our method can reveal more geometry details on real scans, which justifies our capability of handling noise in point clouds.

DeepLS

GridPull

N2NM

Ours

GT

Evaluation on KITTI. Following GridPull [16], we further evaluate our method on KITTI [26] odometry dataset (Sequence 00, frame 3000 to 4000), which contains about 13.8 million points, which are split into 15 chunks. We reconstruct each of them and concatenate them together for visualization. We compare our method with the latest methods SAP [70] and GridPull [16]. As shown in Fig. 8, our method is robust to noise in real scans, successfully generalizes to large-scale scenes, and achieves visual-appealing reconstructions with more details.

Evaluation on Paris-rue-Madame. Following N2NM [52], we further evaluate our method on Paris-rue-Madame [75], which contains much noises. We split the 10 million points into 50 chunks each of which is used to learn a neural implicit function. We compare our method with LIG [37] and N2NM [52]. For LIG [37], we produce the results for each chunk with released pretrained models. For N2NM [52], we overfit on all chunks until convergence. As shown in Fig. 9, we achieve better performance over LIG [37] and N2NM [52] in large-scale surface reconstruction, which highlight our advantages in reconstructing complete and detailed surfaces from noisy scene point clouds.

4.3 Ablation Studies

We conduct ablation studies on the ABC dataset [22] to justify each module of our method.

Embedding Size. We evaluate our performance on different sizes of embedding c . We try several sizes { 128 , 256 , 512 } to infer the signed distance functions from a noisy point cloud. The numerical comparison in Tab. 8 shows that the optimal

result is obtained with a size of 256 . Deviations from this value, either longer or shorter dimensions, leads to worse results with the current number of training samples.

Prior. We conduct experiments to explore the importance of data-driven based prior. We first replace our learned embedding c and parameter

with randomly initialized embedding and parameter, or only replace c with randomly initialized embedding. As shown in Tab. 9, The degenerated result of 'Without Prior' and 'Without Embed' indicates that directly inferring implicit functions without our prior or learned embedding makes it difficult to accurately learn the surfaces of the noisy point clouds, and also slows the convergence. Then we fix the learned parameters and only optimize the embedding c , similar to auto-decoding. The results also get worse, as shown in 'Fixed Param'.

Local Region Splitting. We further validate the effectiveness of local region splitting strategies. We employ three different splitting strategies in Tab. 10. We first

split the whole space where the noisy point cloud is located uniformly into multiple voxel blocks, as shown by the result of 'Voxel'. The severely degenerated results indicate that this splitting strategy is even worse than the global method N2NM [52], as it results in many empty voxel blocks. Then we randomly select a point from the noisy point cloud as a center to sample all points within a radius of 0 . 1 as a local region. The result of 'Sphere (Fixed Size)' slightly degenerates due to some of the spheres containing too few points. In contrast, our splitting strategy, as shown by the result of 'Sphere (KNN)', ensures that each local region has enough points to help achieve superior performance.

SAP

GridPull

Ours

LIG

N2NM

Ours

Global and Local. With our learned prior, we compare our performance in global and local mappings with finetuning the priors. We report results obtained with the local noise to noise mapping or the global one during the finetuning. As shown in Tab. 11, the numerical comparison shows that the global

mapping struggles to infer local details from noisy point clouds. Moreover, our local prior also converges faster than the global statistical reasoning.

Local Region Size. We further validate the effectiveness of local region sizes (points number in a local region) in Tab. 12. We use different local region sizes including { 500 , 1000 , 3000 , 5000 } . The results show that 1000 is the best.

SDF initialization. We further validate the effectiveness of different SDF initializations in Tab. 13 and Fig. 10, including random initialization, geometry initialization [1], initialization to a simple square shape, and ours.

We can see our prior can reconstruct more accurate surfaces from single noisy point clouds in much shorter time than any other initializations.

Random

Square

Sphere

Ours

GT

Half Noise

GT

Noise Type. We report our performance with various noise types, i.e., impulse noise, quantization noise, Laplacian noise, and Gaussian noise. Visual comparison in Fig. 12 justifies that we can also handle other types of noise quite well. Moreover, we also tried more challenging cases with nonuniform noises which do not have a zero expectation across a shape, like a shape with only a half of points having noises or a shape with several patches having noises. The result in Fig. 11 shows that

Impulse noise

Quantization noise

Laplace noise

GT

Figure 12: Visual results with different noise types. our method can also handle nonuniform noises well.

Noise Level. We report our performance on point clouds with different levels of noise. As shown in Tab. 14, the noise levels of middle and max come from the ABC dataset [22]. The middle indicates noises

Ours

R

with a variance of 0 . 01 L , where L is the longest edge of the bounding box. The max indicates noises with a variance of 0 . 05 L . Our extreme noise comes with a variance of 0 . 07 L .

The CD L 2 comparison shows that our results slightly degenerate with max and extreme noise, but still outperform N2NM [52]. The visual results in Fig. 13 indicates that our method is more robust to noises even when the noise variance is as large as 7%.

Sparsity. We report the effect of the sparsity of noisy point clouds. We downsample the noisy point clouds to 25% and 50% of their original size to validate the impact of sparsity. The CD L 2 results in Tab. 15 and visual comparisons in Fig. 14 indicate that our method can handle sparsity in noisy point

clouds better than N2NM [52]. Since our data-driven based prior can help to learn a more complete surface and reduce the impacts brought by the sparsity.

10%

Time Consumption. Since our method can handle sparsity and require less time as the point number decreases, we conduct an experiment with downsampled noisy points in Tab. 16. Fig. 14 indicates that we can

work well on much fewer points, and also provide an alternative of improving efficiency.

Optimization.

We visualize the optimization process in Fig. 15. We reconstruct meshes using the neural SDF f learned in different iterations. We

0

500

1000

1500

2000

2500

3000

3500

4000

see that the shape is updated progressively to the ground truth shapes.

5 Conclusion

We propose a method to resolve the key problem in inferring SDFs from a single noisy point cloud. Our method can effectively use a data-driven based prior as an initialization, and infer a neural SDF by overfitting on a single noisy point cloud. The novel statistical reasoning successfully infers an accurate and smooth signed distance field around the single noisy point cloud with the data-driven based prior. By finetuning data-driven based priors with statistical reasoning, our method significantly improves the robustness, the scalability, the efficiency, and the accuracy in inferring SDFs from single point clouds. Our experimental results and ablations studies show our superiority and justify the effectiveness of the proposed modules.

References

Matan Atzmon and Yaron Lipman. SAL: Sign agnostic learning of shapes from raw data. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Matan Atzmon and yaron Lipman. SALD: sign agnostic learning with derivatives. In International Conference on Learning Representations , 2021.

Yizhak Ben-Shabat, Chamin Hewa Koneputugodage, and Stephen Gould. DiGS: Divergence guided shape implicit neural representation for unoriented point clouds. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Yizhak Ben-Shabat, Chamin Hewa Koneputugodage, and Stephen Gould. DiGS : Divergence guided shape implicit neural representation for unoriented point clouds. CoRR , abs/2106.10811, 2021.

Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael J. Black. Dynamic FAUST: Registering human bodies in motion. In IEEE Computer Vision and Pattern Recognition , 2017.

Alexandre Boulch and Renaud Marlet. POCO: Point convolution for surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6302-6314, June 2022.

Alexandre Boulch and Renaud Marlet. Poco: Point convolution for surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020.

Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In European Conference on Computer Vision , 2020.

Pedro Hermosilla Casajus, Tobias Ritschel, and Timo Ropinski. Total denoising: Unsupervised learning of 3d point cloud cleaning. In IEEE International Conference on Computer Vision , pages 52-60, 2019.

Rohan Chabra, Jan Eric Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard A. Newcombe. Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In European Conference on Computer Vision , volume 12374, pages 608-625, 2020.

Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], Stanford University -Princeton University - Toyota Technological Institute at Chicago, 2015.

Chao Chen, Zhizhong Han, and Yu-Shen Liu. Unsupervised inference of signed distance functions from single sparse point clouds without learning priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17712-17723, 2023.

Chao Chen, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Unsupervised learning of fine structure generation for 3D point clouds by 2D projections matching. In IEEE International Conference on Computer Vision , 2021.

Chao Chen, Yu-Shen Liu, and Zhizhong Han. Latent partition implicit with surface codes for 3d representation. In European Conference on Computer Vision , 2022.

Chao Chen, Yu-Shen Liu, and Zhizhong Han. GridPull: Towards scalability in learning implicit representations from 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 18322-18334, 2023.

Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. IEEE Conference on Computer Vision and Pattern Recognition , 2019.

Julian Chibane, Aymen Mir, and Gerard Pons-Moll. Neural unsigned distance fields for implicit function learning. arXiv , 2010.13938, 2020.

Gene Chou, Ilya Chugunov, and Felix Heide. GenSDF: Two-stage learning of generalizable signed distance functions. In Advances in Neural Information Processing Systems , pages 24905-24919, 2022.

Christopher B. Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3D-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, European Conference on Computer Vision , volume 9912, pages 628-644, 2016.

Angela Dai and Matthias Nießner. Neural Poisson: Indicator functions for neural fields. arXiv preprint arXiv:2211.14249 , 2022.

Philipp Erler, Paul Guerrero, Stefan Ohrhallinger, Niloy J. Mitra, and Michael Wimmer. Points2Surf: Learning implicit surfaces from point clouds. In European Conference on Computer Vision , 2020.

Miguel Fainstein, Viviana Siless, and Emmanuel Iarussi. DUDF: Differentiable unsigned distance fields with hyperbolic scaling. arXiv preprint arXiv:2402.08876 , 2024.

Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point set generation network for 3D object reconstruction from a single image. In 2017 IEEE Conference on Computer Vision and Pattern Recognition , pages 2463-2471, 2017.

Qiancheng Fu, Qingshan Xu, Yew-Soon Ong, and Wenbing Tao. Geo-Neus: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction. 2022.

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Computer Vision and Pattern Recognition , 2012.

Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T. Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In International Conference on Computer Vision , 2019.

Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 3789-3799, 2020.

Zhizhong Han, Chao Chen, Yu-Shen Liu, and Matthias Zwicker. DRWR: A differentiable renderer without rendering for unsupervised 3D structure learning from silhouette images. In International Conference on Machine Learning , 2020.

Zhizhong Han, Chao Chen, Yu-Shen Liu, and Matthias Zwicker. ShapeCaptioner: Generative caption network for 3D shapes by learning a mapping from parts detected in multiple views to sentences. In ACM International Conference on Multimedia , 2020.

Zhizhong Han, Xiyang Wang, Yu-Shen Liu, and Matthias Zwicker. Hierarchical view predictor: Unsupervised 3d global feature learning through hierarchical prediction among unordered views. In Proceedings of the 29th ACM International Conference on Multimedia , pages 3862--3871, 2021.

Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Point2mesh: a self-prior for deformable meshes. ACM Transactions on Graphics , 39(4):126, 2020.

Pengchong Hu and Zhizhong Han. Learning neural implicit through volume rendering with attentive depth fusion priors. In Advances in Neural Information Processing Systems , 2023.

Jiahui Huang, Hao-Xiang Chen, and Shi-Min Hu. A neural galerkin solver for accurate surface reconstruction. ACM Trans. Graph. , 41(6), 2022.

Jiahui Huang, Zan Gojcic, Matan Atzmon, Or Litany, Sanja Fidler, and Francis Williams. Neural kernel surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4369-4379, 2023.

Meng Jia and Matthew Kyan. Learning occupancy function from point clouds for surface reconstruction. arXiv , 2010.11378, 2020.

Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser. Local implicit grid representations for 3D scenes. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. SDFDiff: Differentiable rendering of signed distance fields for 3D shape optimization. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Michael M. Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. ACM Transactions on Graphics , 32(3):29:1-29:13, 2013.

Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. In Jennifer G. Dy and Andreas Krause, editors, International Conference on Machine Learning , volume 80, pages 2971-2980, 2018.

Chen-Hsuan Lin, Chaoyang Wang, and Simon Lucey. SDF-SRN: Learning signed distance 3D object reconstruction from static images. In Advances in Neural Information Processing Systems , 2020.

Cheng Lin, Changjian Li, Yuan Liu, Nenglun Chen, Yi-King Choi, and Wenping Wang. Point2Skeleton: Learning skeletal representations from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4277-4286, 2021.

Minghua Liu, Xiaoshuai Zhang, and Hao Su. Meshing point clouds with predicted intrinsic-extrinsic ratio guidance. In European Conference on Computer vision , 2020.

Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. DIST: Rendering deep implicit signed distance function with differentiable sphere tracing. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Shi-Lin Liu, Hao-Xiang Guo, Hao Pan, Pengshuai Wang, Xin Tong, and Yang Liu. Deep implicit moving least-squares functions for 3D reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition , 2021.

Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3D supervision. In Advances in Neural Information Processing Systems , 2019.

William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3D surface construction algorithm. Computer Graphics , 21(4):163-169, 1987.

Shitong Luo and Wei Hu. Differentiable manifold reconstruction for point cloud denoising. In ACM International Conference on Multimedia , pages 1330-1338. ACM, 2020.

Shitong Luo and Wei Hu. Score-based point cloud denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4583-4592, 2021.

Baorui Ma, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Neural-pull: Learning signed distance functions from point clouds by learning to pull space onto surfaces. In International Conference on Machine Learning , 2021.

Baorui Ma, Yu-Shen Liu, and Zhizhong Han. Reconstructing surfaces for sparse point clouds with on-surface priors. In IEEE Conference on Computer Vision and Pattern Recognition , pages 6305-6315, 2022.

Baorui Ma, Yu-Shen Liu, and Zhizhong Han. Learning signed distance functions from noisy 3D point clouds via noise to noise mapping. In International Conference on Machine Learning , pages 23338-23357. PMLR, 2023.

Baorui Ma, Yu-Shen Liu, Matthias Zwicker, and Zhizhong Han. Reconstructing surfaces for sparse point clouds with on-surface priors. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Baorui Ma, Yu-Shen Liu, Matthias Zwicker, and Zhizhong Han. Surface reconstruction from point clouds by learning predictive context priors. In IEEE Conference on Computer Vision and Pattern Recognition , pages 6316-6327, 2022.

Baorui Ma, Yu-Shen Liu, Matthias Zwicker, and Zhizhong Han. Surface reconstruction from point clouds by learning predictive context priors. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Baorui Ma, Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han. Towards better gradient consistency for neural signed distance functions via level set alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17724-17734, 2023.

Aihua Mao, Biao Yan, Zijing Ma, and Ying He. Denoising point clouds in latent space via graph convolution and invertible neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5768-5777, June 2024.

Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. ACORN: adaptive coordinate networks for neural scene representation. CoRR , abs/2105.02788, 2021.

Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3D reconstruction in function space. In IEEE Conference on Computer Vision and Pattern Recognition , 2019.

Zhenxing Mi, Yiming Luo, and Wenbing Tao. SSRNet: Scalable 3D surface reconstruction network. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Mateusz Michalkiewicz, Jhony K. Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders P. Eriksson. Deep level sets: Implicit surface representations for 3D shape inference. CoRR , abs/1901.06802, 2019.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision , 2020.

Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In International Conference on Computer Vision , 2021.

Amine Ouasfi and Adnane Boukhayma. Few 'zero level set'-shot learning of shape signed distance functions in feature space. In European Conference on Computer Vision , 2022.

Amine Ouasfi and Adnane Boukhayma. Robustifying generalizable implicit shape networks with a tunable non-parametric model. In Advances in Neural Information Processing Systems , 2023.

Amine Ouasfi and Adnane Boukhayma. Mixing-denoising generalizable occupancy networks. In International Conference on 3D Vision , 2024.

Amine Ouasfi and Adnane Boukhayma. Unsupervised occupancy learning from sparse point cloud. In IEEE Conference on Computer Vision and Pattern Recognition , 2024.

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In IEEE Conference on Computer Vision and Pattern Recognition , 2019.

Songyou Peng, Chiyu "Max" Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, and Andreas Geiger. Shape as points: A differentiable poisson solver. In Advances in Neural Information Processing Systems , 2021.

Songyou Peng, Michael Niemeyer, Lars M. Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision , volume 12348, pages 523-540, 2020.

Francesca Pistilli, Giulia Fracastoro, Diego Valsesia, and Enrico Magli. Learning graph-convolutional representations for point cloud denoising. In European Conference on Computer Vision , volume 12365, pages 103-118, 2020.

Marie-Julie Rakotosaona, Vittorio La Barbera, Paul Guerrero, Niloy J. Mitra, and Maks Ovsjanikov. Pointcleannet: Learning to denoise and remove outliers from dense point clouds. Computer Graphics Forum , 39(1):185-203, 2020.

Konstantinos Rematas, Ricardo Martin-Brualla, and Vittorio Ferrari. Sharf: Shape-conditioned radiance fields from a single view. In International Conference on Machine Learning , 2021.

Andrés Serna, Beatriz Marcotegui, François Goulette, and Jean-Emmanuel Deschaud. Paris-rue-madame database - A 3D mobile laser scanner dataset for benchmarking urban detection, segmentation and classification methods. In International Conference on Pattern Recognition Applications and Methods , pages 819-824, 2014.

Jaehyeok Shim and Kyungdon Joo. DITTO: Dual and integrated latent topologies for implicit 3d reconstruction. arXiv preprint arXiv:2403.05005 , 2024.

Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems , 2020.

Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3D-structure-aware neural scene representations. In Advances in Neural Information Processing Systems , 2019.

Peng Songyou, Niemeyer Michael, Mescheder Lars, Pollefeys Marc, and Geiger Andreas. Convolutional occupancy networks. In European Conference on Computer Vision , 2020.

Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In IEEE Conference on Computer Vision and Pattern Recognition , 2021.

Jiapeng Tang, Jiabao Lei, Dan Xu, Feiying Ma, Kui Jia, and Lei Zhang. SA-ConvONet: Sign-agnostic optimization of convolutional occupancy networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

Maxim Tatarchenko, Stephan R. Richter, Rene Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do single-view 3D reconstruction networks learn? In The IEEE Conference on Computer Vision and Pattern Recognition , 2019.

Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Carsten Stoll, and Christian Theobalt. PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations. European Conference on Computer Vision , 2020.

Delio Vicini, Sébastien Speierer, and Wenzel Jakob. Differentiable signed distance function rendering. ACM Transactions on Graphics , 41(4):125:1-125:18, 2022.

Jiepeng Wang, Peng Wang, Xiaoxiao Long, Christian Theobalt, Taku Komura, Lingjie Liu, and Wenping Wang. NeuRIS: Neural reconstruction of indoor scenes using normal priors. In European Conference on Computer Vision , 2022.

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Advances in Neural Information Processing Systems , pages 27171-27183, 2021.

Peng-Shuai Wang, Yang Liu, and Xin Tong. Deep octree-based cnns with output-guided skip connections for 3d shape and scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 266-267, 2020.

Ruian Wang, Zixiong Wang, Yunxiao Zhang, Shuangmin Chen, Shiqing Xin, Changhe Tu, and Wenping Wang. Aligning gradient and hessian for neural signed distance function. In Advances in Neural Information Processing Systems , volume 36, pages 63515-63528, 2023.

Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. HF-NeuS: Improved surface reconstruction using high-frequency details. 2022.

Zhen Wang, Shijie Zhou, Jeong Joon Park, Despoina Paschalidou, Suya You, Gordon Wetzstein, Leonidas Guibas, and Achuta Kadambi. Alto: Alternating latent topologies for implicit 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 259-270, 2023.

Francis Williams, Zan Gojcic, Sameh Khamis, Denis Zorin, Joan Bruna, Sanja Fidler, and Or Litany. Neural fields as learnable kernels for 3D reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18500-18510, 2022.

Francis Williams, Teseo Schneider, Claudio Silva, Denis Zorin, Joan Bruna, and Daniele Panozzo. Deep geometric prior for surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition , 2019.

Francis Williams, Matthew Trager, Joan Bruna, and Denis Zorin. Neural splines: Fitting 3D surfaces with infinitely-wide neural networks. In IEEE Conference on Computer Vision and Pattern Recognition , pages 9949-9958, 2021.

Yunjie Wu and Zhengxing Sun. DFR: differentiable function rendering for learning 3D generation from images. Computer Graphics Forum , 39(5):241-252, 2020.

Peng Xiang, Xin Wen, Yu-Shen Liu, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Zhizhong Han. SnowflakeNet: Point cloud completion by snowflake point deconvolution with skip-transformer. In IEEE International Conference on Computer Vision , 2021.

Huizong Yang, Yuxin Sun, Ganesh Sundaramoorthi, and Anthony Yezzi. StEik: Stabilizing the optimization of neural signed distance functions and finer shape representation. In Advances in Neural Information Processing Systems , 2023.

Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Advances in Neural Information Processing Systems , 2021.

Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems , 33, 2020.

Wang Yifan, Shihao Wu, Cengiz Oztireli, and Olga Sorkine-Hornung. Iso-Points: Optimizing neural implicit surfaces with hybrid representations. CoRR , abs/2012.06434, 2020.

Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. MonoSDF: Exploring monocular geometric cues for neural implicit surface reconstruction. 2022.

Sergey Zakharov, Wadim Kehl, Arjun Bhargava, and Adrien Gaidon. Autolabeling 3D objects with differentiable rendering of sdf shape priors. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Wenbin Zhao, Jiabao Lei, Yuxin Wen, Jianguo Zhang, and Kui Jia. Sign-agnostic implicit learning of surface self-similarities for shape modeling and reconstruction from raw point clouds. CoRR , abs/2012.07498, 2020.

Junsheng Zhou, Baorui Ma, Shujuan Li, Yu-Shen Liu, and Zhizhong Han. Learning a more continuous zero level set in unsigned distance fields through level set projection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3181-3192, 2023.

Junsheng Zhou, Baorui Ma, Yu-Shen Liu, Yi Fang, and Zhizhong Han. Learning consistency-aware unsigned distance functions progressively from raw point clouds. In Advances in Neural Information Processing Systems , 2022.

Qian-Yi Zhou and Vladlen Koltun. Dense scene reconstruction with points of interest. ACM Transactions on Graphics , 32(4):112:1-112:8, 2013.

Runsong Zhu, Di Kang, Ka-Hei Hui, Yue Qian, Shi Qiu, Zhen Dong, Linchao Bao, Pheng-Ann Heng, and Chi-Wing Fu. Ssp: Semi-signed prioritized neural fitting for surface reconstruction from unoriented point clouds. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 3769-3778, 2024.

A Appendix

A.1 Limitations

Our method is still limited to too large noises. For noises that corrupted shapes too much, our method still produces bad results. One direction for our future work is to improve our prior, so that we could have a better sense of a shape even under large noises.

A.2 Detailed Comparisons on 3D Scene

We detail our evaluations on each scene in 3D scene dataset in Tab. 17. The comparisons highlight our advantages in each scene.

A.3 More Results

We visualize more surface reconstruction results under ShapeNet [12], ABC [22], Surface Reconstruction Benchmark (SRB) [92], FAMOUS [22], D-FAUST [5] and nuScenes [8] in Fig. 16, Fig. 17, Fig. 18, Fig. 19, Fig. 20 and Fig. 21.

POCO

N2NM

Ours

GT

Input

IMLS

P2S

NeuralPull

OnSurf

N2NM

Ours

GT

Input Point2Mesh

PSR

SIREN

GridPull

Steik

NKSR

N2NM

Ours

Input

IMLS

NeuralPull

LPI

OnSurf

GridPull

N2NM

Ours

GT

ANT

Input

IGR

Point2Mesh

SAP

N2NM

Ours

GT

N2NM

Ours

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in the Appendix A.1.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We describe in Method one of the core contributions of the local noise-tonoise mapping, and although there is no theory or theorem in it, we verify its validity and reasonableness in our experiments.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed information in reproducing our methods in Implementation Details of Section 3.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide our demonstration code as a part of our supplementary materials. We will release our source code, data and sufficient instructions upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the training and test details for shapes and scenes in Section 4. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We report the average performance in terms of several metrics as the experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report our inference time with other methods in the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper conforms in all respects to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the application and potential positive impact of our method in the introduction.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no such risk to the paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use open-source datasets and code under their licence.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method for learning neural signed distance functions (SDFs) from noisy point clouds. This approach integrates the advantages of data-driven and overfitting-based methods to enhance generalization, accuracy, and inference speed. They employ statistical reasoning within local regions to refine the estimation of SDFs without requiring clean point clouds or signed distance supervision.\n\nSTRENGTHS:\n1. This paper successfully combines data-driven and overfitting-based approaches, leveraging their strengths while mitigating their weaknesses. \n2. This paper shows that the method enhances robustness against noise, a common real-world challenge. They effectively use local statistical reasoning to adjust the neural SDFs, improving accuracy in surface reconstruction and denoising tasks, as evidenced by the benchmark results.\n\nWEAKNESSES:\n1. There are a lot of typos in the manuscript, e.g., In the Abstract and Introduction, \"stat-of-the-art\" should be \"state-of-the-art\"; Line 57: \"Learning implicit functions have achieved\" should be \"Learning implicit functions has achieved\"; Line 39: \"overfits an neural network\" should be \" a neural network\", \"Querys\", and so on. In the Abstract, the \"prompt\" might be \"promote\". It would be best to check out all the typos.\n2. Dependency on initial conditions: The performance might heavily rely on the quality of the initial data-driven priors, which might limit the method's effectiveness when such priors are not well-tuned or applicable.\n3. The paper does not extensively discuss the computational demands or scalability when applied to larger datasets (e.g., nuScenes and Waymo) or more complex scenarios.\n\nQUESTIONS:\n1. How does the method perform under limited computational resources? When scaling to larger datasets, is there a significant trade-off between accuracy and computational efficiency?\n2. Can the method maintain its performance across different types of noise or corruption that were not part of the training set?\n3. How does the method perform when such priors are unavailable, or how sensitive is it to the initial conditions set by these priors?\n\nLIMITATIONS:\nSee the \"Questions\" and \"Weaknesses\" sections above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe presented work tackles the problem of reconstructing a shape from a noisy point cloud (PC) into an implicit representation, using the signed distance function (SDF). Recent methods are categorised into i) data-driven approaches that learn a shape prior with a dataset of training shapes, with poor generalization but generally faster, or ii) approaches that overfit on a single PC at a time, with better generalization but slow convergence. The paper proposes a middle ground: finetuning a pre-trained shape prior model to a single PC, using a novel approach that performs \"local statistical reasoning\": supervising local parts of the shape with multiple sampled noisy PC. The method is experimentally validated against multiple baselines, either data-driven or overfitting-based, and on multiple datasets showing better reconstruction accuracy at lower convergence times. A final ablation study shows that the prior and locality of the loss play a role in the method's performance.\n\nSTRENGTHS:\n1. Experiments are extensive: the PC reconstruction is evaluated against multiple baselines (both data-driven or overfitting-based) on various datasets (ShapeNet, ABC, FAMOUS, SRB, and D-FAUST). In nearly all cases and presented metrics, the proposed approach is the best.\n2. I found the locality approach to solving the problem interesting; supervising parts of the shape at a time instead of the full one.\n3. Using a shape prior that is unrelated to the target object (such as ShapeNet prior for FAMOUS reconstruction) also works. This suggests that this approach is applicable even if the target shape is from an unknown category (thus helps for generalization).\n4. The method is fairly simple and clear, which should make it easily usable.\n\nWEAKNESSES:\n1. The main pipeline is fairly simple (fine-tuning a pre-trained model). I believe the locality of the \"statistical reasoning\" is perhaps the main novelty, and the ablation shows that it improves accuracy and convergence time (§ at L.342), but it is not discussed or explained why using a local approach instead of a global one improves convergence and accuracy.\n2. The writing could be improved. More specifically, the experimental results on the different datasets all make more or less the same points and so could be condensed into something more compact. This can free up some space to expand other parts and increase their clarity, e.g., the related works and the \"Neural Signed Distance Function.\" paragraph (L.101).\n3. Throughout the work, the noise is assumed (additive) Gaussian. While this is not a weakness in itself, this still is an assumption that should be explained and made clear:\n   1. Why is a Gaussian noise model good? Are Lidar data, for instance, usually with gaussian noise and what real use-cases does it apply to? Generally speaking, the motivation of the work could be expanded beyond \"reconstructing a PC to an SDF\".\n   2. \"Statistically, the expectation of the zero-level set should have the minimum distance to all the noisy point splitting.\" (L.131-132) This is true assuming Gaussian noise with expectation 0. With other noise types the loss may not perform as well (e.g., \"Text removal\" in Noise2Noise [1]).\n\n[1] Lehtinen et al., Noise2Noise: Learning Image Restoration without Clean Data, ICML 2018.\n\nQUESTIONS:\n1. What is the reasoning behind using the locality in the \"statistical reasoning\" and its better performance?\n2. For the ShapeNet dataset, is it per-category and then averaged or trained on multiple categories combined? (and which ones?) Same questions regarding the SRB/FAMOUS experiments with ShapeNet prior.\n3. What are the PC sizes? It's hard to appreciate the other parameters without this info. What are the datasets sizes in number of training and test shapes?\n4. How does the prior help if it apparently does not need to be from the same dataset as the test shapes (e.g., SRB/FAMOUS experiments)? Is it actually pre-training on any valid SDFs that helps? If so, how do you think it compares to the simple SAL [2] geometric initialization?\n5. Why are the metrics (CD-L2/L1, NC, F1-score) not used consistently through the datasets and results?\n\n[2] Atzmon and Lipman, SAL: Sign Agnostic Learning of Shapes from Raw Data, CVPR 2020.\n\nLIMITATIONS:\nLimitations are briefly mentioned in the appendix regarding noise level. What kind of noise level is considered too big? It might also be worth commenting on the noise types (see Weakness 3.), is the method working only for Gaussian noises?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an implicit surface reconstruction method from point clouds that uses a learned prior to initialize the optimization of a neural SDF. First, a neural SDF generator based on DeepSDF [66] is trained on a dataset of shapes to learn a prior over shapes. Given a point cloud, both the shape code used as input for the generator, as well as the parameters of the neural SDF are then optimized to fit the point cloud, using an approach based on Noise to Noise Mapping (N2NM) [50], but focusing on local patches instead of the full global point cloud. The authors show that this approach performs better than N2NM as well as several state-of-the-art methods in an extensive evaluation on several datasets.\n\nSTRENGTHS:\n- Using a learned prior as initialization for an optimization-based surface reconstruction method like N2NM has not been done before as far as I can tell.\n- Using local patches also seems like a (smaller) contribution to me (the authors show that this contributes to the performance, but it is not very clear to me at this point why it improves performance significantly - this could be improved).\n- Results show consistently better reconstruction accuracy than the state-of-the-art.\n- The evaluation is extensive and mostly well-done, and convincingly shows and advantage over the state-of-the-art.\n\nWEAKNESSES:\n- While the idea of using a learned prior as initialization of an optimization based method has not been done for surface reconstruction as far as I can tell, it is not very surprising that this can be done and is also relatively straight-forward to do. Both stages are essentially the same as existing work [66] and [50], with some deviation from [50] in using local patches. So all-in-all, the technical contribution does not seem huge, although possibly still large enough for acceptance.\n- While local patches do seem to improve performance, it is not fully clear why this is the case, and the exact details of using local patches are missing some details in the exposition, making it unclear exactly how they are implemented.\n- A few recent methods are missing both from the related work and the comparisons (see below).\n- The exposition could use some additional details and clarifications in some parts (see below).\n\nMost of the weaknesses, apart from the first one, can be addressed with text changes to some extent. Given the good performance of the method and the extensive evaluation, I lean towards acceptance.\n\nDetails:\n- The field of surface reconstruction from point clouds is quite vast, so the authors missed a few works:\n  - Neural-Singular-Hessian: Implicit Neural Representation of Unoriented Point Clouds by Enforcing Singular Hessian, Wang et al., TOG 2023\n  - Iterative Poisson Surface Reconstruction (iPSR) for Unoriented Points, Hou et al., TOG 2022\n  - 3DShape2Vecset: A 3D Shape Representation for Neural Fields and Generative Diffusion Models, Zhang et al., TOG 2023\n  - Geoudf: Surface reconstruction from 3d point clouds via geometry-guided distance representation, Ren et al., TOG 2023\n\n- Given good normals, surface reconstruction becomes much easier - the normal computation could be followed by Poisson reconstruction based on the normals for example. Therefore papers to compute oriented normals are relevant, such as:\n  - Orienting Point Clouds with Dipole Propagation, Metzer et al., TOG 2021\n  - Globally Consistent Normal Orientation for Point Clouds by Regularizing the Winding-Number Field, Xu et al., TOG 2023\n  - SHS-Net: Learning Signed Hyper Surfaces for Oriented Normal Estimation of Point Clouds, Li et al., CVPR 2023\n\n- The exposition could use some additional details and clarifications:\n  - The approach for using local patches is missing some details: How many patches are used during optimization? In which order are patches evaluated? One after the other? Or is a random patch chosen in each iteration? This should be clarified.\n  - The difference of the second stage to [50] could use some more discussion. The authors describe the application of [50] to local patches as main difference (as shown in Eq. 3). But since the expectancy in Eq. 3 is taken over random samples near random patches, how is this different than taking the expectancy over random samples near the whole shape (as in [50])? This could use some  discussion. (Or is the SDF actually optimized separately per patch? From reading I did not get this impression.)\n  - [50] also uses a geometric consistency constraint as regularization, although such a regularization is not described by the authors. If it is not used, a discussion why the authors removed it might be useful.\n  - c seems to be initialized to a fixed constant vector: the average of all embeddings in the training set, while the original DeepSDF paper uses a c randomly sampled from a Gaussian. Why did the authors use a different initialization for c here? Did the average empirically perform better? This could be discussed.\n  - In the paragraph starting at line 129, It should be stated clearly that the method described here is the method proposed in [50] (although evaluated at random samples near random local patches, rather than random samples near the whole shape).\n\n- The evaluation is quite extensive, but would benefit from a few improvements:\n  - In the evaluation, it would be good to hear more about how the time used to fine-tune each optimization-based method was chosen. Was each result optimized until convergence? Or otherwise, how was the number of iterations/optimization time chosen?\n  - It would be good to add DeepSDF to the comparison, as it is the closest to the initialization used for the optimization stage, and would show how much the fine-tuning  improves accuracy.\n  - In the ablation, using a large local region seems worse than using a global approach without local regions. This should be discussed.\n\n- References [52] and [53] are duplicates\n\nQUESTIONS:\n- A clarification of the unclear details regarding local patches would help.\n- A clarification of how timings for methods were chosen would help as well.\n\nLIMITATIONS:\nThe authors have not discussed clear limitations unfortunately.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes 3D shape reconstruction given a noisy point cloud using a test-time optimization approach. The paper proposes a new loss function that can work directly on noisy point clouds without the need of ground truth normals or SDF. The approach involves learning a network (DeepSDF) to predict SDFs and later fine-tune the network on a single shape during test. The learned priors reduce the test-time fitting and leads to better reconstruction results.\n\nSTRENGTHS:\n1. The paper proposes a simple and easy to implement approach to 3D reconstruction that works on a bit noisy point clouds.\n2. The better quality results and fast inference has a broad significance in 3D reconstruction.\n\nWEAKNESSES:\n1. In the introduction, the test-time optimization is hailed as a novel thing, however previous methods (including DeepSDF) has already proposed it.\n2. The main contribution is the loss function defined using \"pulling\" operation, which forces the surface defined by the projected points close to the input points be similar to the input point clouds. However, why this loss function should work in a noisy point cloud setting is not well motivated and well explained.\n3. The writing of the method section can be significantly improved. For example, the symbol $f$ seems to represent both the neural network that predicts the signed distance field and the field it self. More about it in the Questions section.\n4. The use of the phrase \"statistical reasoning\"is quite vague. The phrase is used throughout the manuscript but does not convey anything meaningful.\n5. It is not clear whether the baselines in the experiment section are also trained on noisy data to make them robust.\n6. Comparison with Points2Surf [21] method is missing. I think it work quite well on shapes in Figure 4.\n7. What does \"extreme noise\" mean in the Table 13? What does it translate to in percentage of the scale of the scene.\n8. How do you achieve 100% sparsity in Table 14? Does it imply no points?\n9. Noise level of 0.5% is quite low to claim that the method is robust.\n\nQUESTIONS:\n1. Please improve the notations in the method section. The vectors should be bold. Use different symbols for parameterized function and output.\n2. The motivation of the loss function is not clear. Perhaps a diagram would enhance the understanding for a reader.\n\nLIMITATIONS:\n1. Limitations regarding pose variations and incomplete data (holes) are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}