MultiPull: Detailing Signed Distance Functions by Pulling Multi-Level Queries at Multi-Step

Authors: takeshi  noda
chao  chen
weiqi  zhang
xinhai  liu
yu-shen  liu
zhizhong  han
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-05 03:34:56

Abstract: Reconstructing a continuous surface from a raw 3D point cloud is a challenging task. Latest methods employ supervised learning or pretrained priors to learn a signed distance function (SDF). However, neural networks tend to smooth local details due to the lack of ground truth signed distnaces or normals, which limits the performance of learning-based methods in reconstruction tasks. To resolve this issue, we propose a novel method, named MultiPull, to learn multi-scale implicit fields from raw point clouds to optimize accurate SDFs from coarse to fine. We achieve this by mapping 3D query points into a set of frequency features, which makes it possible to leverage multi-level features during optimization. Meanwhile, we introduce optimization constraints from the perspective of spatial distance and normal consistency, which play a key role in point cloud reconstruction based on multi-scale optimization strategies. Our experiments on widely used object and scene benchmarks demonstrate that our method outperforms the state-of-the-art methods in surface reconstruction.

Main paper: 
 introduction: 
 Reconstructing surfaces from 3D point clouds is an important task in computer vision. It is widely used in various real-world scenarios such as autonomous driving, 3D scanning and other downstream applications. Recently, using neural networks to learn signed distance functions from 3D point clouds has made huge progress [1, 2, 3, 4, 5, 6, 7, 8 ]. An SDF represents a surface as the zero-level set of a 3D continuous field, and the surface can be further extracted using the marching cubes algorithm [9] . In supervised methods [10, 11, 12, 13] , a continuous field is learned using signed distance supervision. Some methods employ multi-level representations [14, 15] , such as Fourier layers and level of detail (LOD) [16, 17] , to learn detailed geometry. However, these methods require 3D supervision, including ground truth signed distances or point normals, calculated on a watertight manifold. To address this issue, several unsupervised methods [18, 19, 20, 21, 22] were proposed to directly infer an SDF by overfitting neural networks on a single point cloud without requiring ground truth signed distances and point normals. They usually need various strategies, such as geometric constraints [18, 19, 20] and consistency constraints [22, 23] , for smoother and more completed signed distance field. However, the raw point cloud is a highly discrete approximation of the surface, learning b) and (c), SDFs are learned from a point cloud by optimizing multi-level query points at multi-step. At each step, we optimize query points at one level with frequency features at this specific level as conditions. This enables the network to progressively recover coarse-to-fine geometry details.SDFs directly from the point cloud is often inaccurate and highly ambiguous. This makes it hard for the network to learn accurate SDFs on local details, resulting in over-smooth reconstruction.To address this issue, we propose MultiPull, to learn an accurate SDF with multi-scale frequency features. It enables network to predict SDF from coarse to fine, significantly enhancing the accuracy of the predictions. Furthermore, to optimize the SDF at different scales simultaneously, we introduce constraints on the pulling process. Specifically, given query points sampled around 3D space as input, we use a Fourier transform network to represent them as a set of Fourier features. Next, we design a network that can leverage multi-scale Fourier features to learn an SDF fields from coarse to fine. To optimize the signed distance fields with multi-scale features, we introduce a loss function based on gradient consistency and distance awareness. Compared with Level of Detail (LOD) methods [16, 17, 24] , we can optimize the signed distance fields effectively without a need of signed distance supervision, recovering more accurate geometric details. Evaluations on widely used benchmarks show that our method outperforms the state-of-the-art methods. Our contribution can be summarized as follows.• We propose a novel framework that can directly learn SDFs with details from raw point clouds, progressing from coarse to fine. This provides a new perspective for recovering 3D geometry details. • We introduce a multi-level loss function based on gradient consistency and distance awareness, enabling the network to geometry details. • Our method outperforms state-of-the-art methods in surface reconstruction in terms of accuracy under widely used benchmarks. 
related work: 
 Classic methods for geometric modeling [25, 26, 27, 28] have attempted to analyze the geometric modeling of objects, which do not require large-scale datasets. With the advent of extensive and intricate 3D datasets like ShapeNet [29] and ABC [30] , learning-based methods have achieved significant advancements [18, 12, 31, 32, 22, 33, 34, 35, 36, 37, 38] . These approaches learn implicit representations from various inputs, including multi-view images [39, 40, 41, 42, 43, 44] , point clouds [45, 46, 47] , and voxels [48, 49, 50] .Learning Implicit Functions with Supervision. Supervised methods have made significant progress in recent years. These methods leverage deep learning networks to learn priors from datasets or use real data for supervision [10, 11, 51, 52, 53, 54] to improve surface reconstruction performance. Some supervised approaches use signed distances and point normals as supervision, or leverage occupancy labels to guide the network's learning process. In order to improve the generalization ability of neural networks and learn more geometric details, some studies learn geometry prior of shapes through supervised learning.Learning Implicit Functions with Conditions. To alleviate the dependence on supervised information, recent studies focus on unsupervised implicit reconstruction methods. These methods do not require pretrained priors during optimization. For example, NeuralPull (NP) [21] learns SDF by pulling query points in nearby space onto the underlying surface, which relies on the gradient field of the network. CAP [55] further complements this by forming a dense surface by additionally sampling dense query points. GridPull [23] generalizes this learning method to the grid, by pulling the query point using interpolated signed distances on the grid. In addition, some studies explore surface reconstruction more deeply and propose innovative methods, such as utilizing differentiable Poisson solutions [56] , or learning signed [57, 58, 19, 51] or unsigned functions [59, 55] with priors. However, inferring implicit functions without 3D supervision requires a lengthy convergence process, which limits the performance of unsupervised methods on large-scale point cloud datasets. To address this, we propose a fitting-based frequency feature learning strategy that efficiently learns implicit fields without the need for additional supervision.Learning Implicit Functions with LOD. Level-Of-Detail (LOD) models [16, 17, 24] are used to simplify code complexity and refine surface details through the architecture of multi-level outputs.Previous studies have explored multi-scale architectures in various reconstruction tasks. For example, NGLOD [16] uses octree-based feature volumes to represent implicit surfaces, which can adapt to shapes with multiple discrete levels of detail and enable continuous level-of-detail switching through SDF interpolation. MFLOD [17] applies Fourier layers to LOD, which can offer better feasibility in Fourier analysis. However, it is difficult to optimize multi-level features simultaneously to learn 3D shapes. To address this issue, we propose a novel strategy to optimize multi-level frequency features, allowing the network to progressively learn geometric details from coarse to fine. Step Pulling (MSP) module. In (a), we learn Fourier bases h i (Q) from query points Q using the Fourier layer and obtain multi-level frequency features y i through Hadamard product. In (b), using multi-level frequency features from (a) and a linear network LSNN with shared parameters, we calculate the distance(D) of Q i to its corresponding surface target point Q t to predict a more accurate surface. We visualize the predicted SDF distribution map corresponding to the frequency features in (a) and the reconstruction from each step of SDF predictions on the right side of (b).ℒ 𝑟𝑒𝑐𝑜𝑛 ℒ 𝑟𝑒𝑐𝑜𝑛 ℒ 𝑟𝑒𝑐𝑜𝑛 ℒ 𝑔𝑟𝑎𝑑 ℒ 𝑔𝑟𝑎𝑑 ℒ 𝑠𝑢𝑟𝑓 Query Points 𝑄 0 LSNN 𝑄 0 𝑄 1 𝑄 2 𝒴 0 𝒴 1 𝒴𝑁 𝐿-3 𝒴𝑁 𝐿-2 𝒴𝑁 𝐿-1 ℎ0 (𝑄0) ℎ1 (𝑄0) ℎ𝑁 𝐿-3 (𝑄0) ℎ𝑁 𝐿-2 (𝑄0) ℎ𝑁 𝐿-1 (𝑄0) 𝑧1 𝐿 𝑧 𝑁 -3 𝑧 𝑁𝐿-2 𝐿 𝑧 𝑁 -1 
method: 
 Overview. The overview of MultiPull is shown in Fig. 2 . We design a neural network to learn an implicit function f from a single 3D point cloud by progressively pulling a set of query points Q 0 onto the underlying surface, where Q 0 is randomly sampled around the raw point cloud S. Our network mainly consists of two parts as follows.(1) The Frequency Feature Transformation (FFT) Module ( Fig. 2(a) ) aims to convert the query points Q 0 into a set of multi-level frequency features Y = {y i , i ∈ [0, N L -1]}. The key insight for introducing frequency features lies in a flexible control of the degree of details. (2) The Multi-Step Pulling (MSP) Module (Fig. 2(b) ) is designed to predict f with coarse-to-fine details under the guidance of frequency features Y . At the i-th step, we pull Q i to Q i+1 using the predicted signed distances s i = f (Q i , y i ) and the gradients at Q i , according to its feature y i . To this end, we constrain query points to be as close to their nearest neighbor point on S.We introduce a neural network to learn frequency features Y from point clouds. The network manipulates input Q 0 through several linear layers to obtain an initial input z 0 and a set of Fourier basis h i (Q 0 ), i ∈ [0, N L -1], formulated as follows.EQUATIONwhere ω i and ϕ i are the parameters of the network, and N L is the number of layers of the network.To effectively represent the expression of the raw input in the frequency space, we choose the sine function as the activation function and employ the Hadamard product to compute the intermediate frequency feature output. Since the Hadamard product allows the representation of frequency components as the product of two feature inputs, denoted as a and b, which can be formulated as:EQUATIONThrough Eq. ( 2), we can calculate the frequency component z i of h i (Q 0 ), and then obtain the output y i of the i-th layer, formulated as:EQUATIONwhere ⊙ indicates the Hadamard product, W i , b i are parameters of the network.Frequency networks based on the Multiplication Filter Network (MFN) [14] typically employ uniform or fixed-weight initialization for network parameters in practice. This approach overlooks the issue of gradient vanishing in deep network layers during the training process, leading to underfitting and making the network overly sensitive to hyperparameter changes. To address this challenge, we propose a new initialization scheme that thoroughly considers the impact of network propagation, aiming at ensuring a uniform distribution of initial parameters. Specifically, we dynamically adjust initial weights, which can be formulated as:EQUATIONwhere N L and η are the number of layers and the parameters of the network, respectively. We leverage the standard deviation as the initialization range to ensure that the parameters in Eq. ( 3) are within a reasonable range. As shown in Fig. 3 , we compared the parameter distributions of different linear layers. The initialization scheme based on MFN results in gradient vanishing and small activations in deeper linear layers. In contrast, our initialization scheme ensures that the parameters of each linear layer follow a standard normal distribution. In Fig. 2 (b), we demonstrate our idea of learning an accurate implicit function f with multiple frequency features. Given a set of frequency feature Y , we use frequency features y i in Y as the input along with query points Q i for the MSP module. We follow NP [21] to construct initial query points and calculate the stride and direction of Q i at i-th step for pulling it to the target surface point. Furthermore, we use the direction of the gradient as ∇f (Q i , y i ) and signed distance f (Q i , y i ) for the pulling, where ∇f (Q i , y i ) represents the fastest increase in signed distance in 3D space, pointing away from the surface direction. Therefore,Q i = Q i-1 -f (Q i-1 , y i-1 ) • ∇f (Q i-1 , y i-1 )/ ∥ ∇f (Q i-1 , y i-1 ) ∥ 2 .For each step of pulling the query points Q i , it corresponds to a nearest point q i on the surface, and the distance between query points and surface points can be described asD i = ||Q i -q i || 2 2 .Based on this, we initiate the optimization by pulling query points Q i the target points q i progressively. Therefore, we can obtain the combined loss L pull under optimal conditions:EQUATIONwhere I is the step of moving operation. However, optimizing all query points accurately through this equation alone is challenging when merely constraining surface points. This is because the query points Q i may be located across multiple spatial scales with inconsistent gradient directions, indicating that simultaneous optimization becomes challenging. Consequently, some outlier points may not be effectively optimized. Additionally, for sampling points near target points, some surface constraints are required to enable the network to accurately predict their corresponding zero levelset to avoid optimization errors. Therefore, we will further advance Eq. ( 5) from the perspectives of distance constraints, gradient consistency, surface constraints in Sec. 3.3 to enhance network performance.Distance-Aware Constraint. Inspired by FOCAL [60] , we design a novel constraint with distance-aware attention weights α to ensure that the network pays more attention to the optimization of underfitting query points in space and optimizes the SDFs simultaneously. This allows query points at different distances from the surface to be optimized properly, and assigns higher attention weights for outlier and underlying points:EQUATIONwhere α 1 and α 2 are calculated from D 1 , D 2 by the softmax activation, γ is a scaling coefficient we set to 2 by default. Here, we only consider 3 steps, which is a trade-off between performance and efficiency.Consistent Gradients. We additionally introduce consistency constraints in the gradient direction. This loss encourages neighboring level sets to keep parallel, which reduces the artifacts off the surface and smooths the surface. We add a cosine gradient consistency loss function to encourage the gradient direction at the query points to keep consistent with the gradient direction at its target point on the surface, which aims to improve the continuity of the gradient during the multi-step pulling. We use Q 1 , Q 2 and Q 3 to represent the query points that have been continuously optimized by the multiple steps. We take the one with the lowest similarity score to measure the overall similarity.EQUATIONwhere L ∇ (Q i ) represents the loss of cosine similarity between query points Q and target surface points q.Surface Constraint. We introduce the surface constraint for the implicit function f , aiming to assist the network in approaching the zero level-set on the surface at final step. Hence, we constrain the f (Q I , y I ) approaches zero at the final step:EQUATIONJoint Loss Function. Overall, we learn the SDFs by minimizing the following loss function L.EQUATIONwhere β and δ are balance weights. In the subsequent ablation experiments, we validated the effectiveness of different loss functions. 
experiments: 
 In this section, we evaluate the performance of MultiPull in surface reconstruction by conducting numerical and visual comparisons with state-of-the-art methods on both synthetic and real-scan datasets. Specifically, in Sec. 4.1, we experiment on synthetic shape datasets with diverse topological structures. Furthermore, in Sec. 4.2, we report our results across various scales on real large-scale scene datasets. Meanwhile, we consider FAMOUS as the verification dataset in the ablation studies to compare the effectiveness of each module in MultiPull in Sec. 4.3.Datasets and Metrics. For the single shape surface reconstruction task, we perform evaluation on multiple datasets including ShapeNet [29] , FAMOUS [10] , Surface Reconstruction Benchmark (SRB) [45] Thingi10K [61] and D-FAUST [62] . We conduct validation experiments on 8 subcategories within the ShapeNet dataset, while the remaining datasets are experimented on the complete dataset. For metric comparison, we leverage L1 and L2 Chamfer Distance CD L1 and CD L2 , Normal Consistency (NC), and F-Score as evaluation metrics.ShapeNet.We evaluate our approach on the ShapeNet [29] according to the experimental settings of GP [23] . We compared our methods with methods including ATLAS [63] , DSDF [51] , NP [21] , PCP [64] , GP [23] , as shown in Tab. 1. We report CD L2 , NC and F-Score metrics for ShapeNet, where we randomly sample 10,000 points on the reconstructed object surface for evaluation. MultiPull outperforms the state-of-the-art methods. Compare to previous gradient-based methods in Fig. 4 , our method performs better by revealing more local details of these complex structures. We provide detailed results in Appendix C. FAMOUS. We evaluate the performance of our method on the FAMOUS dataset according to the experimental settings of PCP [64] and NP [21] . Our method demonstrates superiority over recent approaches, including GP [23] , PCP [64] , GenSDF [65] , FGC [66] , NP [21] , and IGR [57] . As shown in Tab. 2, we compared the recent methods using CD L2 and NC metrics, and our method exhibits outstanding performance. To demonstrate the effectiveness of our method in reconstruction accuracy, we visualize the error-map for comparison in Fig. 5 . Compare to the the state-of-art methods, our method has better overall reconstruction accuracy (bluer). SRB. We validate our method on the real scanned dataset SRB, following the experimental settings of VisCo [67] and GP [23] . In Tab. 3, we compared our approach with recent methods including P2M [68] , SAP [56] , NP [21] , BACON [15] , CAP [55] , GP [23] . We use CD L1 and F-Score to evaluate performance , and we surpass all others in terms of these metrics. As depicted in Fig. 6 , our method excels in reconstructing more complete and smoother surfaces.D-FAUST. We evaluate our method on the D-FAUST dataset with SAP [56] settings. As indicated in Tab. 4, we compared our approach with recent methods including IGR [57] , SAP [56] , GP [23] .Our method excels in CD L1 , F-Score and NC. As illustrated in Fig. 7 , compared to other methods, our approach demonstrates superior accuracy in recovering human body shapes.Thingi10K. We assess the performance of our approach on the Thingi10K dataset, following the experimental setup of SAP [56] . We compared our approach with recent methods including IGR [57] , SAP [56] , BACON [15] , GP [23] . As indicated in Tab. 5, our method surpasses existing methods across in CD L1 , F-Score and NC metrics. As illustrated in Fig. 8 , our method can reconstruct surfaces with more accurate details. Datasets and Metrics.For the scene reconstruction task, we validate our method on the 3DScene [69] and KITTI [70] datasets to assess the performance on large-scale datasets. We keep the same evaluation metrics as those used for shape reconstruction in Sec. 4.1.3DScene. In accordance with the experimental settings of PCP [64] , we compared our approach with recent methods including ConvOcc [39] , NP [21] , PCP [64] and GP [23] . We report the evaluation results of CD L1 , CD L2 and NC, and compared our method with the latest approaches listed in Tab. 6. As illustrated in Fig. 9 , our method outperforms prior-based methods and overfitting based methods. KITTI. We validate our method on the large-scale scanned point cloud dataset KITTI [70] , which contains 13.8 million points. As shown in Fig. 10 , our approach is capable of reconstructing more complete and accurate surfaces compared to the GP method [23] . GP struggles to reconstruct continuous surfaces such as walls and streets, whereas our method achieves a more detailed reconstruction of objects at various scales in real scanned scenes. It demonstrates that our method is robust when handling point cloud with various scales.Effect of Frequency Layers. We denote the j-th layer of the frequency network as L j , a specific combination of frequency feature layers can be formulated as{L i , L j , L k }, where {i, j, k} ∈ [1, N L -1].We evaluate the effectiveness of the frequency transformer layers in Tab. 7 with CD L2 and NC, replacing the frequency network with linear layers results in a decrease in the performance of the CD L2 and NC metrics. The performance of using only one layer(L 4 ) surpasses linear layers.With the increase of the frequency layers, {L 4 ,L 6 ,L 8 } produces best results.  
conclusion: 
 We propose a novel method to learn detailed SDFs by pulling queries onto the surface at multi-step. We leverage the multi-level features to predict signed distances, which recovers high frequency details. Through optimization, our method is able to gradually restore the coarse-to-fine structure of reconstructed objects, thereby revealing more geometry details. Visual and numerical comparisons show that our approach demonstrates competitive performance over the state-of-the-art methods. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification:We use the open-sourced datasets under their licenses. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Our network consists of two main parts: frequency feature transformation and multi-step pulling modules in Fig 2 (a ) and (b), respectively. For frequency feature transformation, we transform the raw point cloud into frequency features M , where M is initialized to 256. Same for the multi-step pulling module, we train a linear sequence neural network (LSNN) with shared parameters and we fix intermediate layer output dimension to 512. In the construction of query points, we establish the corresponding pairs between query points and their nearest points on surfaces. Specifically, we follow NP [21] to construct 40 queries for each point of the point cloud, the construction of these query points follows a Gaussian distribution. During the reconstruction process, we use the Marching Cubes algorithm [27] to extract the mesh surface.During the training process, we do optimization in 40,000 iterations, with an average time of 24 minutes for single-object reconstruction. We utilize a single NVIDIA RTX-3090 GPU for both training and testing.Effect of Frequency Features. To further validate the superiority of frequency features, we exclude multi-step pulling, and only use single-layer frequency feature for performance verification against linear layers. We note the frequency feature in the i-th layer as L i . We compared the CD L2 and NC of specific layers(L 2 , L 4 , L 6 , L 8 ) with the linear layers. As shown the Tab. 11 the performance of frequency features at different layers is superior to the linear layers, and with an increase in the number of layers, higher-level frequency conditions enhance the network's performance. Effect of Initialization Strategies. We compared our initialization strategy with random initialization and MFN-based method (BACON [15] ) as example. We compared the metrics of these initialization methods in Tab. 12, which shows that combining random or BACON initialization with our approach does not yield satisfactory results.To further demonstrate the advantages of our initialization method, we visually compared SDF with random initialization and BACON initialization strategies. As shown in the Fig. 11 , our method significantly outperforms other initialization methods in terms of convergence speed. In addition, our reconstruction results also indicate that a reasonable initialization method can enable the network to learn more accurate signed distance field. We compared the results with the same iterations and the final results under the default settings for different methods (Final) in Fig. 12 , these failed reconstructions based on MFN demonstrate the instability of parameter initialization. Effect of Parameters on Networks. We compared the parameter quantities of the methods listed in Tab. 13 below. It shows that the parameter number of PCP [64] is the largest among all the three methods, while NP [21] has the least parameters. To further investigate the performance of networks with the similar amount of parameters, we increase the parameters of NP and MultiPull to match PCP. The comparison in the Tab. 14 indicates that both NP and MultiPull show the improved performance. This demonstrates that more parameters are beneficial to improve the performance, but Design of L grad .We further discuss the design of L grad and effectiveness on performance. We use minimum (min) as the baseline and compare it with using the average (avg). As shown in Tab. 16, using L grad (avg) to calculate the similarity of query points at different time steps results in a slight increase in CD error. In contrast, L grad (min) achieves a similar level of similarity but better performance in CD metrics. Therefore, we calculate minimum of the gradient similarities as a constraint to prevent significant deviations in the moving direction during training, making the network more sensitive to changes in gradient direction. Comparison Details for 3DScene. We also provide detailed metrics for single scenes in 3DScene dataset. We evaluate it by CD L1 , CD L2 and NC. As shown in the Tab. 22, our approach achieves the best performance across all scenes. We propose a method that approximates the accurate signed distance field through multi-step optimization, achieving more precise results. However, there is still room for further optimization in terms of time and computational efficiency as shown in Tab. 8 and Tab. 14. In future work, we will continue to explore how to integrate multi-resolution (such as NGLOD [16] and Instant-NGP [71] ) features effectively to balance computational efficiency and accuracy.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide our demonstration code as a part of our supplementary materials. We will release the source code, data and instructions upon acceptance. Guidelines:• The answer NA means that paper does not include experiments requiring code. We discuss the applications and potential impacts of our method in the introduction and contribution sections. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper proposes MultiPull, a multi-scale / multi-step pulling approach to reconstruct a mesh surface from point cloud.  MultiPull trains a neural network to learn a SDF by using multi-scale frequency features. Experiments on several datasets shows that compared to baseline methods, the proposed method has higher reconstruction accuracy and lower errors.

Reviewers are split in their ratings on this work, with three indicating acceptance (ZVf1,VUUa,NaoM) and one voting for reject (uTM5).

The main criticism from reviewer uTM5 is that the proposed method is a combination of pulling-based SDF Neural-Pull [18] and the multiplicative Fourier level of detail MFLOD [14].  In their response, the authors note that it was not trivial to combine the two approaches, as to make use of features from MFLOD and pulling operation from NeuralPull, details of how to initialize and the design of multi-step pulling module was required.  The AC also notes that novel combination of techniques (whether well-known or not) can be valuable (as also noted in the NeurIPS 2024 reviewer guidelines).  As most reviewers find the work to be interesting and there does not appear to be significant flaws with the evaluation, the AC believe this work would be of interest to the NeurIPS community and recommends acceptance.

The authors are encouraged to follow reviewer suggestions to improve the paper:
- Include additional experiments and ablations provided during rebuttal
- Improvements to writing (see detailed comments from VUUa) including adding recent relevant work, discussion of design choices, improve clarity of exposition.
- Please proofread for typos and grammar.  Some (non-exhaustive) examples (in addition to the suggestions from VUUa) are given below:
   - Reword copied sentence (L82-83)   
   - Equations
      - Improve notation (see R-ZVf1 comments)
      - L138: Use "\cdot" not "\times"
      - Improve typesetting (use `\sin', '\cos' for sin/cos, '\text{pull}', '\text{recon}', etc for words in equations)
   - Correct number in last row of Table 7 (ZVf1)
   - Line 2: "Latest" => "the latest" (or just "Recent")
   - Line 4: "distnaces" => "distances"
   - Figure 1: "Setp 2" => "Step 2"
   - Table 1: "ShapNet" => "ShapeNet"
   - Line 211: "compare" => "compared"

Reviews: review: paper_summary: paper_summary: This paper proposes to learn multi-scale implicit fields from 3D point clouds for accurate optimization of SDFs in a coarse-to-fine manner. The spatial query points are first mapped to frequency features through the FFT module. Then the MSP module is designed to exploit the multi-level frequency features to progressively pull the query points towards to target surface. Various experiments and ablation studies demonstrate the effectiveness of the proposed learning approach.
strength_weakness: strength_weakness: The major weakness of this work is its limited novelty. In general, I think the proposed techniques are just an incremental combination of the pulling-based SDF learning approach [18] (Neural-Pull) and the multiplicative fourier learning approach [14] (MFLOD), although with some marginal modifications in order to combine them. The authors seem to have intentionally overlooked the introduction and detailed discussion of these two types of highly related works. Hence, reviewers that are not very familiar with the two lines of works may overestimate the value and technical novelty of the proposed method. Note that I am not saying that combining some existing techniques is not meaningful. What I mean is that, for this paper, the contribution cannot reach the bar of NeurIPS.
questions: questions: In lines 82-83, the authors wrote “The creation of 3D shapes LOD usually depends on mesh decimation, which has difficulty in blending between LODs.” I notice that this particular sentence is directly copied from the paper of MFLOD [14]. Explanations are needed.
limitations: limitations: The presented limitations are vague and less constructive.
paper_summary: paper_summary: This paper proposes MultiPull, a method for reconstructing a surface model by SDF from a 3D point cloud containing only the coordinates of each point. The paper proposes a method for estimating the SDF using the Fourier transform of the surface model predicted from the point cloud only at multi-scale. It also introduces gradient consistency and distance awareness for multi-scale consistency. The key idea is to use multi-scale frequency features, for which we introduce an FFT module corresponding to an encoder and an MSP module corresponding to an implicit decoder. The proposed method shows high reconstruction performance compared to existing methods and is effective on different datasets.
strength_weakness: strength_weakness: - l.138: "\cdot" is better than "\times" to avoid confusion with the cross product.
- My understanding is that there seems to be no guarantee that the scale of the displacement of the query point at each frequency varies with the frequency of interest. Is this understanding correct? Is there some possible constraint on the magnitude of the slope that would be consistent with the frequency of interest? (Or is it that this concern does not exist, since the experimental separation by frequency has been verified?)
- l.166: For the purpose of consistency, would it not be appropriate to use the average or correlation rather than argmin to select only one of Q1, Q2, or Q3? If you have any additional explanation, I would appreciate it.
- In the introduction, it would be desirable to specify whether the proposed method is a test-only optimization (i.e., does not require prior training, such as NeuralPull or SALD) or a method that requires prior training (e.g., DeepSDF). Figure 3 in [*1] clearly shows the position of the proposed method in relation to related research. My understanding is that the paper plots to the same point as NGLOD, and it is preferable to explicitly state that only test time is considered. (If my understanding is incorrect, please correct me.)
- l.243: After considering both performance metrics and time efficiency, we have set Step=3 by default.", can the authors discuss this trade-off? Given the nature of the LOD and the trade-off with the 3D geometry compression ratio, there may be future applications related to mesh compression.

[*1] Francis Williams, Zan Gojcic, Sameh Khamis, Denis Zorin, Joan Bruna, Sanja Fidler, Or Litany. Neural Fields As Learnable Kernels for 3D Reconstruction. CVPR2022.
questions: questions: - l.77: "However, inferring implicit functions without 3D supervision requires a very long convergence process, which limits the performance of unsupervised methods in large-scale point cloud data." I agree with this point from my experiments, but can the author provide a specific reference in the paper?
- l.99: "To this end, we constrain query points to be as close to their nearest neighbor on S." I understood sampling from the nearest neighbor, but did not understand how Q is constructed. Are non-nearest points just ignored?
limitations: limitations: - I agree with the limitations described in the paper. This is a promising direction for future work.
- If there are areas of future work that you would like to see described in Weaknesses and Questions, these may be limitations.
paper_summary: paper_summary: This paper presents a model for reconstructing SDF from point clouds. The proposed approach introduces components, including Frequency Feature Transformation and Multi-Step Pulling, to iteratively refine the reconstructed SDF. The experiments demonstrate that this method performs well across multiple 3D object and scene datasets.
strength_weakness: strength_weakness: 1. Some intuitions behind the methodology are not clearly explained. Specifically, the rationale for using frequency features to represent query points needs further clarification. While the use of multiple levels of query points is understandable, the choice of LSNN at each level and how query points are used in this model require more detailed explanation.
2. Some of the notations are inconsistent. S is used for both raw point cloud in L91 and SDF surface on L99. D is used for both feature dimensions and distance.
3. Minor: The number reported on the last row of Table 7 appears to be wrong.
questions: questions: 1. What’s the intuition for using frequency features and why using LSNN in MSP? 
2. From the visualization, the proposed method seems to produce results with fewer holes and smoother surfaces. Is this due to the model learning better features, or is there a specific loss design different from prior work?
3. Could you compare the MSP using the same features from prior work to demonstrate its performance even without the proposed feature? This would help to convincingly show its effectiveness.
limitations: limitations: No negative social impacts have been identified.
See weakness.
paper_summary: paper_summary: The authors propose a method to reconstruct neural multi-scale SDFs from point clouds using an optimization-based approach. The multi-scale SDF is represented with a network architecture based on previous work [11] and optimized using a new iterative pulling approach, where the SDF for each scale is optimized to pull a query point to the 0-level-set (using the gradient and value of the SDF at the query), starting with a relatively distant query point for the coarsest SDF, and using the updated query position from the result of the previous scale in the next-finer SDF. Losses based on GridPull [20], but adapted to the multi-scale setting are used as objective for the optimization. The authors show that this approach performs better than several exisiting surface reconstruction methods on several synthetic and real-world datasets.
strength_weakness: strength_weakness: - The evaluation is missing some relevant recent methods for optimization-based surface reconstruction. (See details below)
- Some relevant related work is not cited. (See details below)
- The exposition is missing motivation for some design choices and is hard to understand in some parts. (See details below)

Overall, even though the authors missed some related work, results look quite promising and I think the idea is novel and interesting enough to make me lean towards acceptance. The exposition can probably be improved somewhat in the final version.

Details:

- The field of surface reconstruction from point clouds is quite vast, so the authors missed several works, for example (see also related work within these papers to get more missing work):
  -Training-free methods:
    - Neural-Singular-Hessian: Implicit Neural Representation of Unoriented Point Clouds by Enforcing Singular Hessian, Wang et al., TOG 2023
    - Iterative Poisson Surface Reconstruction (iPSR) for Unoriented Points, Hou et al., TOG 2022
  - Learning-based methods:
    - 3DShape2Vecset: A 3D Shape Representation for Neural Fields and Generative Diffusion Models, Zhang et al., TOG 2023
    - Geoudf: Surface reconstruction from 3d point clouds via geometry-guided distance representation, Ren et al., TOG 2023
  - Given good normals, surface reconstruction becomes much easier - the normal computation could be followed by Poisson reconstruction based on the normals for example. Therefore papers to compute oriented normals are relevant, such as:
    - Orienting Point Clouds with Dipole Propagation, Metzer et al., TOG 2021
    - Globally Consistent Normal Orientation for Point Clouds by Regularizing the Winding-Number Field, Xu et al., TOG 2023
    - SHS-Net: Learning Signed Hyper Surfaces for Oriented Normal Estimation of Point Clouds, Li et al., CVPR 2023

- Several design choices are not motivated well enough:
  - It is unclear to me why the SDF evaluated at Q_1 uses a feature y_1 as input that was computed from a query position Q_0, rather than a feature computed at query position Q_1. This is an interesting design choice that could use more discussion. An ablation would also be interesting: how would the method perform when re-evaluating h_0, ...,  h_i at Q_i instead of Q_0 when computing y_i for f(Q_i, y_i) (in that case inputting Q_i a second time to the LSNN might also not be necessary)?
  - In Eq. 6, the design choice of only scaling D_1 and D_2 with the error-based factor, but not D_3 needs to be motivated. Also, as Section 3.2 was written for a general number of layers N_L, it would be good to continue this generality in Section 3.3, if possible. How would Eq. 6 look for a general N_L., for example?
  - In Eq. 7, the design choice for using the minimum over the three levels.
  
- The exposition is hard to understand in some parts and could be improved:
  - Several passages are currently hard to understand and could use clarification. For example: "The is because Q_i  locate at different scales in space,, making it difficult to simultaneously account for the optimization step sizes at different scales with the inconsistent gradient directions caused by continuous movement." Its possible to get a good guess at what the authors mean here, but it requires re-reading a few times and is not 100% clear even then. Partially this seems to be due to bad grammar, partially due to a somewhat convoluted formulation. Generally the paper text could use more passes to avoid these issues.
  - In Section 3.1, it should be mentioned early on that the FFT module is based on (or at least very similar to) MFN [11]. Also, the exact differences between the proposed approach in Section 3.1 and the approach of MFN [11] should be discussed and clarified.
  - In Section 3.3, it would be good to mention that the loss functions are based on GridPull [20]. Also, it would be good to discuss the exact differences to GridPull (I assume the adpation to the multi-scale setting is the main difference).
  - The architecture of the Linear Sequence Neural Network (LSNN) is never described. If it is taken from existing work, this work should be cited, otherwise it needs to be described, at least in Appendix A.
  - In the paragraph starting at line 112 and Eq. 4, it is unclear which weights are initialized according to this strategy (the omega_i in Eq. 1? The W_i in Eq. 3? The offset theta_i and b_i?), and which parameters are plotted in Figure 3 (the L_i are not defined clearly).
  - In Eq. 4, \eta is not defined.
  - Near line 138, the query point update should probably use Q_{i-1} everywhere on the right-hand-side (instead of Q_i, which is not known yet). Or alternatively, Q_{i+1} on the left-hand side.
  - In Eq. 1, I don't follow why the predicted signed distance f(Q_1, y_1) is in the loss. Why should that be optimized to be 0? This should be clarified.
  - in Eq. 1, ", i \in [1,I]" can be removed, since it is redundant with the subscript in the sum
  - In the section starting at line 160, how is the target surface point q determined? This is described in the appendix, but should be shortly mentioned in the main paper as well.
  - In Eq. 7, argmin should probably be min instead. Also, the "1 - ..." in both lines seems odd, should it only be in one of the lines (otherwise the loss would be *higher* the more similar the gradient directions are, instead of lower).
  - Near Eq. 8, the authors mention that L_surf encourages the SDF at the *final* query point to be close to 0, but Eq. 8 does not show an index for Q. Is L_surf applied to Q_{N_L} only or to all Q_i? This should be clarified.
  - Near line 202: Fig. 4.1 should probably be  Fig. 5
  - In Table 7, Step 5 should probably have CD_L2 of 0.0357 (not 0.357)
  - SAL [15] is in the bibliography, but not referenced from Section 2.
  - Citations [45] and [59] are duplicates.

- An inference time comparison between methods should be included.
questions: questions: A clarification of the design choices could help, as well as a discussion of the differences and advantages/disadvantages compared to the missing related work.
limitations: limitations: Limitations in the long computation time are discussed. I don't foresee any particular negative societal impacts.

Comments: title: We are happy to take any questions
comment: Dear reviewer uTM5,

As the reviewer-author discussion period is about to end, can you please let us know if our rebuttal addressed your concerns? If it is not the case, we are looking forward to taking the last minute to make further explanation and clarification.

Thanks,

The authors
Thank you for your reply, I have no further questions at this time.
Thanks for the comments.

Actually, we were using multi-step pulling procedure, that is the end point of previous pulling will be the start point of the next pulling. This is like recursively calling the SDF iteration by iteration, which does not bring too much burden. Moreover, only the first pulling is observed the large movement, while all the following pulling is just like little adjustment. This is because the first pulling will pull most of queries to somewhere by the zero-level set, then the following pulling will not have too much room to move the queries, or the queries will get pulled far away from the zero-level set in another direction. Thus, pulling in the last several steps is still happening in a pretty local area.

Moreover, it did slow down the feature query process if we use pulled queries like Q1 or Q2 to query features. Currently, using Q0 can help us obtain all features needed in the following pretty fast.

We will make sure to include these as a discussion in our revision.

Best,

The authors
Thanks for the detailed replies and the new set of evaluations!

The added comparison to Neural-Singular-Hessian looks promising. This addresses my concern to some extent (although a more complete comparison would obviously be better).

About the frequency features, I understand the author's point of view that choosing the same query point for all features ensures their consistency, but on the other hand, it also requires these features to make predictions about distant more positions, i.e. about query positions Q1, Q2, ..., as well as needing to take the trajectory caused by previous levels into account (i.e. where did Q1 end up given the prediction of the previous level?), which seems like a more difficult non-local task to me. A more in-depth discussion/analysis would be interesting here, but I would consider this to be optional.

The author's response addresses most of my concerns.
I appreciate the polite point-by-point responses from the authors. In particular, I believe that the validity of the paper has been increased by additional results for (2) and (4).

I have no further questions at this time.
Dear reviewers,

As the reviewer-author discussion period is about to end, we are looking forward to your feedback on our rebuttal. Please let us know if our responses address your concerns. We would be glad to make any further explanation and clarification.

Thanks,

The authors
Dear reviewers,

We appreciate your comments and expertise. Please let us know if there is anything we can clarify further. We would be happy to take this opportunity to discuss with you.

Thanks,

The authors