Learning Unsigned Distance Fields from Local Shape Functions for 3D Surface Reconstruction

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 09:29:26

Abstract: Unsigned distance fields (UDFs) provide a versatile framework for representing a diverse array of 3D shapes, encompassing both watertight and non-watertight geometries. Traditional UDF learning methods typically require extensive training on large datasets of 3D shapes, which is costly and often necessitates hyperparameter adjustments for new datasets. This paper presents a novel neural framework, LoSF-UDF, for reconstructing surfaces from 3D point clouds by leveraging local shape functions to learn UDFs. We observe that 3D shapes manifest simple patterns within localized areas, prompting us to create a training dataset of point cloud patches characterized by mathematical functions that represent a continuum from smooth surfaces to sharp edges and corners. Our approach learns features within a specific radius around each query point and utilizes an attention mechanism to focus on the crucial features for UDF estimation. This method enables efficient and robust surface reconstruction from point clouds without the need for shape-specific training. Additionally, our method exhibits enhanced resilience to noise and outliers in point clouds compared to existing methods. We present comprehensive experiments and comparisons across various datasets, including synthetic and real-scanned point clouds, to validate our method's efficacy.

Main paper: 
 introduction: 
 3D surface reconstruction from raw point clouds is a significant and long-standing problem in computer graphics and machine vision. Traditional techniques like Poisson Surface Reconstruction [1] create an implicit indicator function from oriented points and reconstruct the surface by extracting an appropriate isosurface. The advancement of artificial intelligence has led to the emergence of numerous neural network-based methods for 3D reconstruction. Among these, neural implicit representations have gained significant influence, which utilize signed distance fields (SDFs) [2] [3] [4] [5] [6] [7] [8] and occupancy fields [9] [10] [11] [12] to implicitly depict 3D geometries. SDFs and occupancy fields extract isosurfaces by solving regression and classification problems, respectively. However, both techniques require internal and external definitions of the surfaces, limiting their capability to reconstructing only watertight geometries. Therefore, unsigned distance fields [13] [14] [15] [16] [17] [18] [19] [20] have recently gained increasing attention due to their ability to reconstruct non-watertight surfaces and complex geometries with arbitrary topologies.Reconstructing 3D geometries from raw point clouds using UDFs presents significant challenges due to the non-differentiability near the surface. This characteristic complicates the development of loss functions and undermines the stability of neural network training. Various unsupervised approaches [17, 14, 19] have been developed to tailor loss functions that leverage the intrinsic characteristics of UDFs, ensuring that the reconstructed geometry aligns closely with the original point clouds.However, these methods suffer from slow convergence, necessitating an extensive network training time to reconstruct a single geometry. As a supervised method, GeoUDF [15] learns local geometric 2 Related Work Surface reconstruction. Reconstructing 3D surfaces from point clouds is a classic and important topic in computer graphics. The most widely used Poisson method [1, 22] fits surfaces by solving PDEs. These traditional methods involve adjusting the gradient of an indicator function to align with a solution derived from a (screened) Poisson equation. A crucial requirement of these methods is the input of oriented normals. The Iterative Screened Poisson Reconstruction method [23] introduced an iterative approach to refine the reconstruction process, improving the ability to generate surfaces from point clouds without direct computation of normals. The shape of points [24] introduced a differentiable point-to-mesh layer by employing a differentiable formulation of PSR [1] to generate watertight, topology-agnostic manifold surfaces.Neural surface representations. Recently, the domain of deep learning has spurred significant advances in the implicit neural representation of 3D shapes. Some of these works trained a classifier neural network to construct occupancy fields [9] [10] [11] [12] for representing 3D geometries. Poco [12] achieves superior reconstruction performance by introducing convolution into occupancy fields.Ouasfi et al. [25] recently proposed a uncertainty measure method based on margin to learn occupancy fields from sparse point clouds. Compared to occupancy fields, SDFs [2] [3] [4] [5] [6] [7] [8] offer a more precise geometric representation by differentiating between interior and exterior spaces through the assignment of signs to distances. Some recent SOTA methods, such as DeepLS [3] , using volumetric SDFs to locally learned continuous SDFs, have achieved higher compression, accuracy, and local shape refinement.Unsigned distance fields learning. Although Occupancy fields and SDFs have undergone significant development recently, they are hard to reconstruct surfaces with boundaries or nonmanifold features. G-Shell [26] developed a differentiable shell-based representation for both watertight and non-watertight surfaces. However, UDFs provide a simpler and more natural way to represent general shapes [13] [14] [15] [16] [17] [18] [19] [20] . Various methods have been proposed to reconstruct surfaces from point clouds by learning UDFs. CAP-UDF [17] suggested directing 3D query points towards the surface Figure 1 : Pipeline. First, we train a UDF prediction network U Θ on a synthetic dataset, which contains a series of local point cloud patches that are independent of specific shapes. Given a global point cloud P, we then extract a local patch P assigned to each query point q within a specified radius, and obtain the corresponding UDF values U Θ(P , q). Finally, we extract the mesh corresponding to the input point cloud by incorporating the DCUDF [32] framework. with a consistency constraint to develop UDFs that are aware of consistency. LevelSetUDF [14] learned a smooth zero-level function within UDFs through level set projections. As a supervised approach, GeoUDF [15] estimates UDFs by learning local geometric priors from training on many 3D shapes. DUDF [19] formulated the UDF learning as an Eikonal problem with distinct boundary conditions. UODF [20] proposed unsigned orthogonal distance fields that every point in this field can access to the closest surface points along three orthogonal directions. Instead of reconstructing from point clouds, many recent works [27] [28] [29] [30] learn high-quality UDFs from multi-view images for reconstructing non-watertight surfaces. Furthermore, UiDFF [31] presents a 3D diffusion model for UDFs to generate textured 3D shapes with boundaries. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations in the conclusion section (Sec.5).Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Motivation. Distinct from SDFs, there is no need for UDFs to determine the sign to distinguish between the inside and outside of a shape. Consequently, the UDF values are solely related to the local geometric characteristics of 3D shapes. Furthermore, within a certain radius for a query point, local geometry can be approximated by general mathematical functions. Stemming from these insights, we propose a novel UDF learning framework that focuses on local geometries. We employ local shape functions to construct a series of point cloud patches as our training dataset, which includes common smooth and sharp geometric features. Fig. 1 illustrates the pipeline of our proposed UDF learning framework.Smooth patches. From the viewpoint of differential geometry [33] , the local geometry at a specific point on a regular surface can be approximated by a quadratic surface. Specifically, consider a regular surface S : r = r(u, v) with a point p on it. At point p, it is possible to identify two principal direction unit vectors, e 1 and e 2 , with the corresponding normal n = e 1 × e 2 . A suitable parameter system (u, v) can be determined such that r u = e 1 and r v = e 2 , thus obtaining the corresponding first and second fundamental forms asEQUATIONwhere κ 1 , κ 2 are principal curvatures. Without loss of generality, we assume p corresponding to u = v = 0 and expand the Taylor form at this point asEQUATIONDecomposing r uu (0, 0), r uv (0, 0), and r vv (0, 0) along the tangential and normal directions, we can formulate Eq.(2) according to Eq.(1) asEQUATIONwhere o(u 2 + v 2 ) ≈ 0 is negligible in a small local region. Consequently, by adopting {p, e 1 , e 2 , n} as the orthogonal coordinate system, we can define the form of the local approximating surface as which exactly are quadratic surfaces z = 1 2 (κ 1 x 2 + κ 2 y 2 ). Furthermore, in relation to Gaussian curvatures κ 1 κ 2 , quadratic surfaces can be categorized into four types: ellipsoidal, hyperbolic, parabolic, and planar. As shown in Fig. 2 , for differentiable points on a general geometry, the local shape features can always be described by one of these four types of quadratic surfaces.EQUATIONSharp patches. For surfaces with sharp features, they are not differentiable at some points and cannot be approximated in the form of a quadratic surface. We categorize commonly seen sharp geometric features into four types, including creases, cusps, corners, and v-saddles, as illustrated in Fig. 2(b) .We construct these four types of sharp features in a consistent form z = f (x, y) like smooth patchesEQUATIONwhere h can adjust the sharpness of the shape, and k can control the direction of the crease. Synthetic training dataset. We utilize the mathematical functions introduced above to synthesize a series of point cloud patches for training. As shown in Fig. 3 , we first uniformly sample m points {(x i , y i )} m i=1 within a circle of radius r 0 centered at (0, 0) in the xy-plane. Then, we substitute the coordinates into Eq.(4-5) to obtain the corresponding z-coordinate values, resulting in a patch P = {p m i=1 }, where p i = (x i , y i , z(x i , y i )). Subsequently, we randomly collect query points {q i } n i=1 distributed along the vertical ray intersecting the xy-plane at the origin, extending up to a distance of r 0 . For each query point q i , we determine its UDF value U(q i ), which is either |q i | for sharp patches. Noting that for patches with excessively high curvature or sharpness, the minimum distance of the query points may not be the distance to (0, 0, z(0, 0)), we will exclude these patches from our training dataset. Overall, each sample in our synthetic dataset is specifically in the form of {q, P, U(q)}. We perform supervised training on the synthesized dataset which is independent of specific shapes.The network learns the features of local geometries and utilizes an attention-based module to output the corresponding UDF values from the learned features. After training, given any 3D point clouds and a query point in space, we extract the local point cloud patch near the query, which has the same form as the data in the training dataset. Consequently, our network can predict the UDF value at that query point based on this local point cloud patch.For a sample {q, P = {p i } m i=1 , U(q)}, we first obtain a latent code f p ∈ R lp related to the local point cloud patch P through a Point-Net [34] F p . To derive features related to distance, we use relative vectors from the patch points to the query point, V = {p i -q} m i=1 , as input to a Vectors-Net F v , which is similar to the Point-Net F p . This process results in an additional latent codef v ∈ R lv .Subsequently, we apply a cross-attention module [35] to obtain the feature codes for the local geometry,EQUATIONwhere we take f p as the Key-Value (KV) pair and f v as the Query (Q). In our experiments, we set l p = l v =EQUATIONwhere we set λ d = 0.01 in our experiments.Evaluation. Given a 3D point cloud P for reconstruction, we first normalize it to fit within a bounding box with dimensions ranging from [-0.5, 0.5]. Subsequently, within the bounding box space, we uniformly sample grid points at a specified resolution to serve as query points. Finally, we extract the local geometry P p for each query point by collecting points from the point cloud that lie within a sphere of a specified radius centered on the query point. We can obtain the predicted UDF values by the trained network U Θ * (q, P q ), where Θ * represents the optimized network parameters. Note that for patches P p with fewer than 5 points, we set the UDF values as a large constant. Finally, we extract meshes from the UDFs using the DCUDF model [32] .4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We will definitely make our code publicly available one day, but not at this moment.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiment setup: 
 Datasets. To compare our method with other state-of-the-art UDF learning approaches, we tested it on various datasets that include general artificial objects from the field of computer graphic. Following previous works [30, 17, 14] , we select the "Car" category from ShapeNet [21] , which has a rich collection of multi-layered and non-closed shapes. Furthermore, we select the real-world dataset DeepFashion3D [36] for open surfaces, and ScanNet [37] for large outdoor scenes. To assess our model's performance on actual noisy inputs, we conducted tests on real range scan dataset [38] following the previous works [17, 14] .Baselines. For our validation datasets, we compared our method against the state-of-the-art UDF learning models, which include unsupervised methods like CAP-UDF [17] , LevelSetUDF [14] , and DUDF [19] , as well as the supervised learning method, GeoUDF [15] . We trained GeoUDF independently on different datasets to achieve optimal performance. display obvious visual artifacts such as small holes and non-smoothness. We thus apply DCUDF [32] , the state-of-art method, to each baseline model , extracting the surfaces as significantly higher quality meshes. Since our method utilizes DCUDF for surface extraction, we adopt it as the default technique to ensure consistency and fairness in comparisons with the baselines. Our method achieves stable results in reconstructing various types of surfaces, including both open and closed surfaces, and exhibits performance comparable to that of the SOTA methods. Noting that DUDF [19] requires normals during training, and GeoUDF utilizes the KNN approach to determine the nearest neighbors of the query points. Although DUDF and GeoUDF achieve better evaluations, they are less stable when dealing with point clouds with noise and outliers. Noise & outliers. To evaluate our model with noisy inputs, we added Gaussian noise N (0, 0.0025) to the clean data across all datasets for testing. The middle three rows in Fig. 5 display the reconstructed surface results from noisy point clouds, and Tab. 2 also presents the quantitative comparisons. It can be observed that our method can robustly reconstruct smooth surfaces from noisy point clouds.Clean Noise Outlier CD ↓ F1 ↑ CD ↓ F1 ↑ CD ↓ F1 ↑ method F 1 0.005 F 1 0.01 F 1 0.005 F 1 0.01 F 1 0.Additionally, we tested our method's performance with outliers by converting 10% of the clean point cloud into outliers, as shown in the last three rows of Fig. 5 . To further demonstrate the robustness of our method, we conducted experiments on point clouds with higher percentage of outliers. Our framework is able of reconstructing reasonable surfaces even with 50% outliers. We also tested the task on point clouds containing both noise and outliers. Please refer to Fig. 9 in the Appendix for the corresponding results.Real-world scanned data. Dataset [38] provide several real-world scanned point clouds, as illustrated in Fig. 6 (Left), we evaluate our model on the dataset to demonstrate the effectiveness. Our approach can reconstruct smooth surfaces from scanned data containing noise and outliers. However, our model cannot address the issue of missing parts. This limitation is due to the local geometric training strategy, which is independent of the global shape. Additionally, we conduct tests on large scanned scenes to evaluate our algorithm, as shown in Fig. 6 (Right).Efficiency.As a supervised UDF learning method, our approach has a significant improvement in training efficiency compared to GeoUDF [15] . Patch radius. During the evaluation phase, the radius r used to find the nearest points for each query point determines the size of the extracted patch and the range of effective query points in the space. As shown in Fig. 7 , we analyzed the impact of different radii on the reconstruction results.An excessively small r will generate artifacts, while an overly large r will lose many details. In our experiments, we generally set r to 0.018. Denoising module. Our framework incorporates a denoising module to handle noisy point clouds.We conducted ablation experiments to verify the significance of this module. Specifically, we set λ d = 0 in the loss function Eq. ( 7) to disable the denoising module, and then retrained the network.As illustrated in Fig. 8 , we present the reconstructed surfaces for the same set of noisy point clouds with and without the denosing module, respectively. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We introduce all the training and test details in the main text and appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We provide various evaluation metrics about our method.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide the related information in the experimental section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We strictly adhere to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Our method may be applied to 3D reconstruction in daily life, demonstrating significant social value.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The original owners of all code, data, and models in our paper are properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Justification: There is no new assets attached to our paper. We will make our code and data public once paper is accepted.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.  
 
 Appendix:  


Decision: The submissions got two positive recommendations and two negative recommendations. The reviewers were concerned about the sensitivity to parameters, computational complexity, and method justification. The authors addressed some of these concerns, but some reviewers were still concerned about the limited scenarios where the proposed method can work. During the AC and reviewers discussion period, the AC and the reviewers also had intense discussions on the novelty over some previous methods which also used patches from meshes rather than the ones produced by equations. According to these discussions, it seems that the proposed method is not novel enough. Regarding this,  the AC did not see any explanations on the difference to the related works  in the authors' reply to reviewer 1t3W’ s questions either. Per these, the AC made a decision to reject this submission. The decision was approved by the SAC.

Reviews: review: paper_summary: paper_summary: The authors propose a method for open surface reconstruction from 3D point clouds. They train a network to predict unsigned distance functions (UDFs) from point cloud patches using only synthetic data of quadratic surfaces. Evaluation shows that the trained network generalizes well to other complex patterns and is more resilient to noise when reconstructing 3D surfaces from point clouds.
strength_weakness: strength_weakness: I have three main concerns: potential biases with the synthetic training data, the evaluation scheme, and the applicability in practice due to the patch radius.


## Potential Biases with Synthetic Training Data: 

The use of primitive geometrical patches as training data for a UDF regressor might introduce biases. The observation that any local geometries can be approximated by quadratic surfaces is only valid at a very fine resolution, which requires dense point clouds to observe reliably. It is unclear how many patches have been synthesized and whether they provide a good approximation of universal geometrical primitives. Some analyses would be helpful here: for example, using the ShapeNet car dataset, cropping all local patches from each car, and finding the closest synthesized one to check for approximation errors. Are there any patterns with sufficiently high approximation errors? Can we perform these analyses at different resolutions and see how they correlate with surface reconstruction?

## Evaluation Scheme

The testing data is simulated to match the scenarios the method is designed for: the point cloud is quite dense, and artificial noise is added similarly to the training data. There is no quantitative evaluation for real-world scanned data. 

## How sensitive is the method to different patch radii?

The value of 0.018 is oddly specific. I suspect that with a larger value of r, the method will generate overly smooth surfaces (as shown in Figure 6 - right), and with a smaller value of r, it will generate holes due to the point cloud not being dense enough. Overall, there is an inherent issue with this trade-off that may not be resolvable with this approach. Detailed experiments varying the patch radius and analyzing the impact on reconstruction quality would be helpful.

## Additional Concerns

- Ablation studies on the network architecture are missing. I am unsure about the roles of the two branches and the cross-attention mechanism. There is a potential issue with the reliance on Point-Net for embedding computation. Is this network pre-trained on other datasets? If so, we should be careful with the claim of using only synthetic data, as pre-training on real data could influence the results.

- It also seems that the method could be quite slow. The authors should include a speed test to provide insights into the computational efficiency of the proposed approach. Evaluating the method's runtime on different hardware setups and for varying point cloud sizes would give a clearer picture of its practical applicability.
questions: questions: Main questions I would like the answers for:
- Analyses on using the synthetic patches to approximate the real local 3D geometries.
- Speed test.
- Quantitative results on real-world scanned data, such as 3D-Scene dataset.
limitations: limitations: The authors acknowledged that the proposed method cannot handle incomplete point cloud. However, it is unclear what how resilient it is when dealing with this.
paper_summary: paper_summary: The paper proposes a novel training strategy to learn Unsigned Distance Fields from local shapes. The idea is to train the model on a dataset of point cloud patches characterized by mathematical functions representing a continuum from smooth surfaces to sharp edges and corners. Although trained only on synthetic surfaces, it demonstrates a remarkable capability in predicting UDFs for a wide range of  surface types. The method is evaluated on “Car” category of ShapeNet dataset in addition to DeepFashion3D, and ScanNet datasets. Furthermore, the paper shows results  on scans from real range scan dataset , in addition to ablation studies highlighting the robustness of the method to noise and outliers.
strength_weakness: strength_weakness: - **Inference time**: While the method is better in terms of data storage space, data preparation time and training time, no comparison regarding the  inference time is provided.
- **Patch radius**: The patch radius used at test time is a crucial hyper-parameter for the method. A discussion about how to set this parameter and how it depends on the point cloud density/size would strengthen the paper.
questions: questions: -  What the inference time of the method compared to other baselines ?
- How to set the patch radius for new shapes with different pointcloud densities and sizes ?
limitations: limitations: The authors adequately addressed the limitations of the method.
paper_summary: paper_summary: The paper proposes an approach to reconstruct 3D surfaces from point clouds, using unsigned distance fields. The proposed approach consists in training a specific neural network architecture to predict UDF values from local point cloud patches, which can be triangulated using UDF meshing methods. The paper mathematically analyses the possible local patches that can appear, smooth and sharp, and trains the proposed architecture with them. Once trained, the neural network is queried for UDF values from a point cloud, and it computes them by extracting a local point cloud patch for each query point and applying the information learned during training. A denoising module is employed to reduce the impact of noise and outliers. Experiments are carried out on ShapeNet cars and DeepFashion3D, synthetically sampling point clouds from the dataset meshes and reconstructing surfaces using a number of baselines, an extracting meshes using DCUDF. Results show that the performance of the proposed method is not the strongest on clean data, but it is so in the presence of noise and/or outliers. Experiments on real-world point clouds are shown qualitatively.
strength_weakness: strength_weakness: I believe there are a few weaknesses in the paper, mostly regarding clarity, architecture validation and experiments. I list them in no particular order.
1) The paper proposes a complex architecture, consisting of two branches, a cross-attention module, fully connected layers and a denoising module. The latter is only qualitatively justified in Fig.8, but the rest of the architecture is not validated in any way. A (quantitative) experimental evaluation of why this architecture is suitable for the task would be needed, in my opinion, as well as the intuition that led the researchers to arrive to this final architecture.
2) The cross-attention mechanism is unclear to me: the two branches (points and vectors net) output a latent code. What is the meaning of a cross-attention module on a single input token (actually one as K-V, one as Q)?
3) In the experiments, a few details are not specified, for example the query grid resolution and the number of points used for the Chamfer distance.
4) A time evaluation is completely missing. The paper claims "Our method is computationally efficient" in the introduction, which is partially justified by the comparatively better training times and storage requirements with respect to GeoUDF, but there is no evaluation of the time required by the method to reconstruct a surface, compared to the other methods. Notice also that DCUDF is an extremely slow meshing algorithm, so the evaluation should be performed both with it and without it.
5) The paper claims "superior performance in surface reconstruction from both synthetic point clouds and real scans, even in the presence of noise and outliers". The experiments however show no superiority in the absence of noise and outliers. Moreover, there are no quantitative experiments on real scans, making it impossible to evaluate whether the noise robustness on synthetic noise also translates to real data. Additionally, the qualitative experiments shown on real data lack comparison with the baselines. Thus, in general, I find the experimental section incomplete and not fully convincing.
questions: questions: Relating to some of the weaknesses, I would ask the authors:
1) Could you clarify the role and mechanism of the cross-attention module in the architecture?
2) Could you specify the missing details? (grid resolution, number of points for Chamfer distance)
3) If you already have them in the logs or if time allows, could you provide the timings for your methods and the baselines?
4) Why are surfaces from real data not evaluated quantitatively?
5) In the conclusions, addressing the limitation of the proposed method on incomplete point clouds, you mention the possibility to readily integrate your framework with other methods, but it is unclear to me how this could be possible. Could you provide a brief explanation?
limitations: limitations: A limitation of the method on completing incomplete surfaces has been correctly addressed and disclosed. 
The limited performance on clean data is shown in the experiments, but not acknowledged in the claims in the introduction.
A quantitative assessment of the performance on real scans and of the time required to reconstruct the surface are missing, making it harder to fully evaluate the possible limitations of the method with respect to the claims.
paper_summary: paper_summary: This paper introduces a new method to learning unsigned distance functions. Specifically, the method leverages local shape priors which brings geometry priors and is also able to handle noises and outliers. The results demonstrate that the proposed method outperforms previous baselines, especially in the corrupted situations.
strength_weakness: strength_weakness: 1. The idea is straightforward to learn local shape functions as the prior for global shape reconstruction. Similar ideas are proposed in SDF-based methods. I would like to see more insight in the differences to the previous methods and also the importance of introducing the local geometric priors for UDF learning. 
Local Implicit Grid Representations for 3D Scenes (CVPR 2020)
Surface Reconstruction from Point Clouds by Learning Predictive Context Priors (CVPR 2022)
Deep local shapes: Learning local sdf priors for detailed 3d reconstruction (ECCV 2020)
2. The visualization is not quite convincing in the geometry details. For example, the reconstructions of real-scans in Fig.6 seems worse than the results of other methods in their papers. The local shape functions is expected to produce reconstruction with more details and sharper edges, but the scene reconstructions lack details. 
3. Also, some reconstructions are too fat, which in my opinion, is caused by the inaccurate UDF near the zero-level set, since the method use DCUDF for UDF meshing.
questions: questions: More references on the local shape functioning and UDF learning will be helpful.

Local Implicit Grid Representations for 3D Scenes (CVPR 2020)
CAP-UDF: Learning Unsigned Distance Functions Progressively from Raw Point Clouds with Consistency-Aware Field Optimization (TPAMI 2024)

Can the proposed method handle large scale scenes like KITTI?
limitations: limitations: Please refer to the strengths and weaknesses above.

Comments: title: Thank You for Your Feedback
To continue and supplement
Response to authors
comment: Dear Reviewers,

Thank you for taking the time to review our responses and for providing additional feedback. We would like to take this opportunity to highlight the unique contributions of our paper.

Unlike existing supervised learning methods for UDFs, which are often tailored to specific training datasets and may require re-training for new datasets, LoSF-UDF does not exhibit such dataset dependence. Once trained on our synthetic datasets, it can be applied directly to new datasets without modification. Our method involves only one model-dependent parameter—the size of local patches. With the default setting, we tested our approach on five diverse datasets: DeepFashion3D, ShapeNetCars, scanned data from SRB and threedscans, and scene point clouds, all of which yielded good experimental results.

If the default setting leads to suboptimal results on new datasets, users can easily fine-tune this hyperparameter through a simple linear scan, without the need for re-training. Given LoSF-UDF's high runtime performance, enhanced robustness against noise and outliers, dataset independence, and potential for integration with unsupervised methods, we believe it can be applied to a wide range of real-world applications.

Thank you once again for your valuable feedback.

Sincerely,
The Authors
Following our previous discussion, here we provide a concrete example to demonstrate that the parameter fine-tuning process for LoSF-UDF is both rapid and straightforward. Below, we consider the Bunny model at 5 different resolutions (10K, 15K, 20K, 30K, and 50K) and present LoSF-UDF's reconstruction results across various settings of the hyperparameter $r$:

|··Radius··|····Bunny\_10k····|····Bunny\_15k····|····Bunny\_20k····|····Bunny\_30k····|····Bunny\_50K····| 
|----------|----------|----------|----------|----------|----------|
|··0.012···|······Failure·····|······Failure·····|·0.029/0.113/0.23·|·0.447/0.729/0.39·|·0.601/0.891/0.52·| 
|··0.018···|·0.101/0.251/0.30·|·0.628/**0.902**/0.53·|·0.816/**0.979**/0.64·|·0.802/**0.998**/0.72·|·0.880/**0.999**/0.81·|
|··0.024···|·0.537/**0.997**/0.77·|·0.514/0.992/0.87·|·0.599/0.994/0.92·|·0.615/0.992/0.98·|·0.73/0.998/1.09·|
|··0.036···|·0.407/0.793/1.02·|·0.425/0.700/1.09·|·0.499/0.822/1.19·|·0.600/0.953/1.82·|·0.671/0.982/2.39·| 

In this table, we report the F-scores ($F^{0.005}$ and $F^{0.01}$, where higher scores indicate better quality of the reconstruction results) and the running time required for patch extraction and UDF inference (minutes) for each density and radius setting. For each resolution (column), the best F-scores are highlighted using boldface fonts to easily identify the most effective settings of radius $r$. 


For a very small radius of $r=0.012$, the method was unable to produce valid reconstruction results for the lower resolutions at 10K and 15K densities. The failure is due to the challenge of representing local features when there are extremely few points within the specified radius. The default setting of \( r=0.018 \) yields satisfactory results for inputs with high and medium densities. However, this setting proves inadequate for the lowest density level of 10K points, where it results in fragmented meshes. At this sparse resolution, each patch with a radius of \( r=0.018 \) encompasses merely 5.75 points on average. Adjusting the radius to \( r=0.024 \) increases the average number of points per patch to 9.82, thereby enhancing the reconstruction accuracy for this lower resolution. While increasing the radius further does raise the number of points per patch, performance worsens because the disparity between actual geodesic disks and synthetic patches becomes too pronounced at larger radii. Typically, for a given model with a fixed resolution, the relationship between performance and the parameter $r$ follows an inverted-U-shaped curve. Consequently, **employing a linear search on a few representative models should efficiently determine the optimal hyperparameter $r$ for a new dataset.** We will include both visual results and performance-$r$ plots to illustrate the impact of these parameter adjustments.
Thank you for your feedback. We acknowledge that the quality of LoSF-UDF's reconstruction results depends on the model-dependent hyperparameter $r$, which specifies the size of local patches. However, we do not consider this dependency to be an inherent flaw. If the default setting of $r$ proves suboptimal for a new dataset, our method allows users to fine-tune this parameter using a few representative models. Subsequently, the pre-trained network can be re-run on these models **without the need for re-training**. Because of the high efficiency of LoSF-UDF, this fine-tuning process is both rapid and straightforward. 

It is also important to note that the reliance on a model-dependent hyperparameter is not unique to our method. For instance, GeoUDF (ICCV 2023) employs a hyperparameter $k$ to identify the $k$-nearest neighbors required for computing weights, which represent the contribution of each neighbor. Similarly, POCO (CVPR 2022) also adopts a model-dependent parameter for searching local neighborhoods. To our knowledge, there are no supervised SDF/UDF learning methods that are entirely free of model-dependent hyperparameter. 

Moreover, as demonstrated in the rebuttal, LoSF-UDF can also be effectively combined with an unsupervised method, serving as an initialization. In such scenarios, it is not necessary to obtain a highly accurate result from LoSF-UDF, which diminishes the importance of fine-tuning the parameter $r$.
Thanks for the response. I will maintain my original score as borderline accept.
Dear reviewers,

As the reviewer-author discussion period is about to end by 8.13, please take a look at other reviewers' reviews and authors' rebuttal at your earliest convenience. It would be great if you could ask authors for more clarification or explanation if some of your concerns are not addressed by the rebuttal.

Thanks,

AC
Again, I still find the choice of the radii value oddly specific. I think it is extremely crucial to the method but the current analysis doesn't answer me very well. It does not simply relate the "satisfactory reconstruction" but it hints when the main idea works and when it doesn't. I believe that with different value, the synthetic data will have to be generated differently to better approximate the "universal set". However, if it is too large, no simple function can generate those and if it is too small, the point cloud has to be extremely dense for the method to become relevant.

Overall, this is an inherent issue of this approach that makes me find it not very useful.
Thank you for your questions. 

**Q: The role of cross attention.**
In our network, the outputs of Points-Net and Vectors-Net are indeed one-dimensional vectors, specifically $\mathbf{f}_p\in\mathbb{R}^{64}$ and $\mathbf{f}_v\in\mathbb{R}^{64}$ in our experiments. The Cross-Attn module plays a crucial role in effectively fusing these two feature sets.
We begin by detailing how the Cross-Attn module processes these vectors. We obtain the query $Q$, key $K$, and value $V$ through the following linear transformations:
$$Q=\mathbf{W}_q\cdot\mathbf{f}_v,\,\,K=\mathbf{W}_k\cdot\mathbf{f}_q,\,\,V=\mathbf{W}_v\cdot\mathbf{f}_q,$$
where $\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v\in\mathbb{R}^{64\times64}$ are learnable weight matrices. Then, we compute attention scores as
$$S=\text{Softmax}(\frac{Q K^T}{\sqrt{d}}),$$
where $d=64$ is the dimension. The attention scores quantify the correlation between $\mathbf{f}_p$ and $\mathbf{f}_v$, with $\mathbf{f}_p$ encapsulating local patch features and $\mathbf{f}_v$ providing distance information. The final feature vector $\mathbf{f}_G$ is obtained by applying these weighted attention scores $S$ to $V$. This method has proven effective for fusing $\mathbf{f}_p$ and $\mathbf{f}_v$. Our supplementary ablation study (Fig.3 Ablation (b)) contrasts this with a direct concatenation of $\mathbf{f}_p$ and $\mathbf{f}_v$, followed by processing through two fully connected layers to produce $\mathbf{f}_g$. The results indicate that direct concatenation yields poor reconstruction performance. 

Although both Points-Net and Vectors-Net could output multiple vector features as a set to be processed through cross-attention, similar to the approaches used in most point cloud segmentation and shape analysis tasks, we opt for a simpler method. Given that we are learning point cloud information from local patches, which contain relatively fewer features, a single-dimensional feature vector is sufficient for our purposes.

**Q: Siren Network.** 
In our response, we discussed integrating our unsupervised method with unsupervised methods to enhance geometric details in reconstructed surfaces. This extension highlights our method's contribution and adaptability. Both CAP-UDF and LevelSetUDF utilize coordinate-based MLPs to predict UDF values, leveraging loss functions such as normal consistency and projection loss to supervise training. Given the proven efficacy of SIREN networks in SDF-based reconstruction tasks, we chose SIREN as our backbone for conducting  integration tests. We initialized the SIREN network using outputs from our method and then proceeded with unsupervised learning. This integration strategy demonstrates that our method can provide a better initialization, leading to improved stability and efficiency in UDF learning. We carried out a comparative experiment on three point clouds shown in Fig.5 in the one-page PDF. We used the same SIREN network and loss functions across tests but differentiated between random initialization and initialization using our method's output. For Tennyson and Horse Head, SIREN with random initialization failed to converge, resulting in poor shape reconstruction. For Gargoyle, while SIREN did converge, we observed that SIREN initialized with our method not only converged faster but also exhibited superior performance in terms of CD and F-scores. We will include detailed implementation on integrating LoSF-UDF with SIREN in revised manuscript.
| Tests | Time(min) | CD($\times 100$) | $F^{0.005}$ | $F^{0.01}$ |
|----------|----------|----------|----------|----------|
| Initialization w/ our method | $\leq$10 | 0.012 | 0.728 | 0.908 |
| Random initialization | 15$\sim$20 | 0.595 | 0.654 | 0.882 |
Thank you for your responses.

I still do no understand two things:

1) The role of cross attention: my understanding was that the points-net extracts one feature per patch and so does the vectors-net. In this case, as I mentioned in my review, I do not understand the role of cross attention. In your comment you seem to suggest that the points-net extract more than one feature. Is that so?

2) When the Siren network is used to reconstruct the surface, it achieves better results than the proposed method. Is this so even for a vanilla Siren network (i.e. without your initialization)? It should be included in the baselines in the paper.
Hi reviewers,

Thanks a bunch for all your hard work as reviewers for NeurIPS 2024.

The discussion period between reviewers and authors has started. Make sure to check out the authors' responses and ask any questions you have to help clarify things by 8.13.

Best,
AC