Shape As Points: A Differentiable Poisson Solver
Authors: Songyou Peng
Chiyu Max Jiang
Yiyi Liao
Michael Niemeyer
Marc Pollefeys
Andreas Geiger
Venue: NeurIPS 2021 Oral (2021)
Keywords: Point Clouds
Shape Representation
3D Reconstruction
Poisson Equation
Implicit Representations
Differentiable

Abstract: In recent years, neural implicit representations gained popularity in 3D reconstruction due to their expressiveness and flexibility. However, the implicit nature of neural implicit representations results in slow inference times and requires careful initialization. In this paper, we revisit the classic yet ubiquitous point cloud representation and introduce a differentiable point-to-mesh layer using a differentiable formulation of Poisson Surface Reconstruction (PSR) which allows for a GPU-accelerated fast solution of the indicator function given an oriented point cloud. The differentiable PSR layer allows us to efficiently and differentiably bridge the explicit 3D point representation with the 3D mesh via the implicit indicator field, enabling end-to-end optimization of surface reconstruction metrics such as Chamfer distance. This duality between points and meshes hence allows us to represent shapes as oriented point clouds, which are explicit, lightweight and expressive. Compared to neural implicit representations, our Shape-As-Points (SAP) model is more interpretable, lightweight, and accelerates inference time by one order of magnitude. Compared to other explicit representations such as points, patches, and meshes, SAP produces topology-agnostic, watertight manifold surfaces. We demonstrate the effectiveness of SAP on the task of surface reconstruction from unoriented point clouds and learning-based reconstruction.

Shape As Points: A Differentiable Poisson Solver
Songyou Peng!.2
Chiyu "Max" Jiang"
Yiyi Liao
Michael Niemeyer
Mare Pollefeys
Andreas Geiger
ETH Zurich
Max Planck Institute for Intelligent Systems, Tübingen University of Tübingen
*Microsoft
Abstract
In recent years, neural implicit representations gained popularity in 3D reconstruc- tion due to their expressiveness and flcxibility. However, the implicit nature of neural implicit representations results in slow inference time and requires care- ful initialization. In this paper, we revisit the classic yet ubiquitous point cloud representation and introduce differentiable point-to-mesh layer using differen- tiable formulation of Poisson Surface Reconstruction (PSR) that allows for GPU- accelerated fast solution of the indicator function given oriented point cloud. The differentiable PSR layer allows us to efficiently and differentiably bridge the explicit 3D point representation with the 3D mesh via the implicit indicator field, enabling end-to-end optimization of surface reconstruction metrics such as Cham- fer distance. This duality between points and meshes hence allows to represent shapes as oriented point clouds, which are explicit, lightweight and expressive. Compared to neural implicit representations, is more interpretable, lightweight, and accelerates inference time by one order magnitude. Compared to other explicit representations such points, patches, and meshes, SAP produces copology-agnostin demonstrate the effectiveness of SAP the task of surface reconstruction from unoriented point clouds and learning-basee reconstruction.
Shape-As-Points (SAP) model
watertight manifold surfaces. We
Introduction
Shape representations are central to many of the recent advancements in 3D computer vision and computer graphics, ranging from neural rendering (41,45,48,55558 to shape reconstruction [10,26,40 52, 70]. While conventional representations such as point clouds and meshes are efficient and well-studied, they also suffer from several limitations: Point clouds are lightweight and easy to obtain, but do not directly encode surface information. Meshes, the other hand, are usually restricted to fixed topologies. More recently, neural implicit representations [10, 50,5) have shown promising results for representing geometry due to their flexibility in encoding varied topologies, and their easy integration with differentiable frameworks. However, as such representations implicitly encode surface information, extracting the underlying surface is typically slow they require numerous network evaluations in 3D space for extracting complete surfaces using marching cubes [10,40 50,, along rays for intersection detection in the context of volumetric rendering (45,47, 49,447,47700 70..
In this work, we introduce novel Poisson solver which performs fast GPU-accelerated Differentiable Poisson Surface Reconstruction (DPSR) and solves for an indicator function from an oriented point cloud in few milliseconds. Thanks the differentiablility of our Poisson solver, gradients from loss on the output mesh loss on the intermediate indicator grid can be efficiently backpropagated to update the oriented point cloud representation. This differential bridge between points, indicator
Work done while UC Berkeley. Corresponding authors.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Points
Voxels m
Meshes [61|
Patches [20]
Implicits [40]
SAP
GT
Representations
[17]
(Ours)
Efficiency Grid Eval Time (128")
n/a
n/a
0.33s
0.012s
Priors
Easy Initialization
X
Watertight
X
x
Quality
No Seff-intersection
n/a
n/a
Appology-Agnostie
Table 1: Overview of Different Shape Representations, Shape-As-Points produces higher quality geometry compared to other explicit representations [11,17,00 61] and requires significantly less inference time for extracting geometry compared neural implicit representations [40].
functions, and meshes allows to represent shapes oriented point clouds. We therefore call this shape representation Shape-As-Points (SAP). Compared to existing shape representations, Shape-As- Points has the following advantages (see also Table 1):
Efficiency: SAP has a low memory footprint as it only requires storing a collection of oriented point samples at the surface, rather than volumetric quantities (voxels) large number of network parameters for neural implicit representations. Using spectral methods, the indicator field can be computed efficiently (12 ms at 128 resolution"), ), compared of neural implicit networks (330 ms using [40] at the same resolution). Accuracy: The resulting mesh can be generated at high resolutions, is guaranteed to be watertight, free from self-intersections Initialization: is easy to initialize SAP with a given geometry such
the typical rather slow query time
and also topology-agnostio template shapes noisy observations. In contrast, neural implicit representations harder initialize, except for few simple primitives like spheres [1]. See supplementary for more discussions.
To investigate the aforementioned properties, perform set of controlled experiments. Moreover, we demonstrate state-of-the-art performance in reconstructing surface geometry from unoriented point clouds in two settings: optimization -based setting that does not require training and is applicable to a wide range of shapes, and a learning- based setting for conditional shape reconstruction that is robust to noisy point clouds and outliers. In summary, the main contributions of this work are:
We present Shape-As-Poine a novel shape representation that is interpretable. lightweight, and yields high-quality watertight meshes low inference times.
The core of the Shape-As AineeAs-Pointts representation is versatile, differentiable and generalizable Poisson solver that can be used for range of applications.
We study various properties inherent to the Shape-As-Paie representation, including inference time, sensitivity to initialization and togology-agnostic representation capacity.
We demonstrate state-of-the-art reconstruction results from noisy unoriented point clouds at a significantly reduced computational budget compared existing methods.
Code is available at http://gthhbbcommtttooooosssssss points.
Related Work
2.1 3D Shape Representations
3D shape representations are central to 3D computer vision and graphics. Shape representations can be generally categorized as being either explicit or implici.. Explicit shape representations and learning algorithms depending on such representations directly parameterize the surface of the geometry, either point cloud [17, 38, 53, 54, 64, 67], parameterized mesh [22, 24, 27, 61] or surface patches [2.20, 37,43, 65, 68.. Explicit representations are usually lightweight and require few parameters represent the geometry, but suffer from discretization, the difficulty represent
OO average. our method requires 12 ms for computing a 128 indicator grid from 15K points on a single NVIDIA GTX 1080Ti GPU. Computing 256 indicator grid requires 140 ms.

---

watertight surfaces (point clouds, surface patches), or are restricted to pre-defined topology (mesh). Implicit representations, in contrast, represent the shape as level set of a continuous function over a discretized voxel grid 114,2533, 25,33,66, 66] or more recently parameterized as neural network, typically referred to as neural implicit functions 500. Neural implicit representations have been successfully used to represent geometries of objects [10, 16, 18,35, 40, 44,47,50,55 57,5,9 62, 63] and scenes [8,26, 34, 46, 57]. Additionally, neural implicit functions are able to represent radiance fields which allow for high-fidelity appearance and novel view synthesis [39,45], However, extracting surface geometry from implicit representations typically requires dense evaluation of multi-layer perceptrons, either on volumetric grid or along rays, resulting in slow inference time. In contrast, SAP efficiently solves the Poisson Equation during inference by representing the shape as an oriented point cloud.
2.2
Optimization- 3D Reconstruction from Point Clouds
Several works have addressed the problem of inferring continuous surfaces from a point cloud. They tackle this task by utilizing basis functions, set properties of the points, or neural networks. Early works in shape reconstruction from point clouds utilize the convex hull or alpha shapes for reconstruction [15]. The ball pivoting algorithm [5] leverages the continuity property of spherical balls of given radius. One of the most popular techniques, Poisson Surface Reconstruction (PSR) [28, 22), solves the Poisson Equation and inherits smoothness properties from the basis functions used in the Poisson Equation. However, PSR is sensitive to the normals of the input points which must be inferred using separate preprocessing step. In contrast, our method does not require any normal estimation and is thus more robust to noise. More recent works take advantage of the continuous nature of neural networks as function approximators to fit surfaces to point sets [19, 23, 42, 65]. However, these methods tend to be memory and computationally intensive, while our method yields high-quality watertight meshes in a few milliseconds.
2.3 Learning-based based 3D Reconstruction from Point Clouds
Learning- based approaches exploit training set of 3D shapes to infer the parameters of a recon- struction model. Some approaches focus on local data priors [2,26] which typically result in better generalization, but suffer when large surfaces must be completed. Other approaches learn object- level [33,40, 50] or scene-level priors [12, 13, 26, 52]. Most reconstruction approaches directly recon- struct meshed surface geometry, though some works 21,311 first predict point set normals to subsequently reconstruct the geometry via PSR [28,29] However, such methods fail to handle large levels of noise, since they are unable move points or selectively ignore outliers. In contrast, end-to-end approach is able to address this issue by either moving outlier points to the actual surface or by selectively muting outliers either by forming paired point clusters that self-cancel or reducing the magnitude of the predicted normals which controls their influence on the reconstruction.
3 Method
At the core of the Shape-As-Poine representation is a differentiable Poisson solver, which can be used for both optimization-base based and learning-basee based surface estimation. We first introduce the Poisson solver in Section 3.1. Next, we investigate two applications using our solver: optimization- based 3D reconstruction (Section 3.2) and learning-basee based 3D reconstruction (Section 3.3).
3.1 Differentiable Poisson Solver
The key step in Poisson Surface Reconstruction [28, 29] involves solving the Poisson Equation. Let x E R3 denote spatial coordinate and n E IR' denote its corresponding normal. The Poisson
Equation arises from the insight that set consisting of point coordinates and normals {p = (c. n)} can be viewed as samples of the gradient of the underlying implicit indicator function x xxx that describes the solid geometry. We define the normal vector field as a superposition of pulse functions v(x) Eie. m.) 6(x c, n,), where 5(x, n) {n if x and 0 otherwise). By applying the divergence operator, the variational problem transforms into the standard Poisson equation:
V2x = V Vx=V -x=v... V.v
In order to solve this set of linear Partial Differential Equations (PDEs), we discretize the function values and differential operators. Without loss of generality, we assume that the normal vector field and the indicator function X are sampled at uniformly spaced locations along each dimension. Denote the spatial dimensionality of the problem to be d. Without loss of generality, consider
3

---

the three dimensional case where n : the point normal field v E Raxd the gradient operator V R Raxd the divergence operator (V-) Rnnd > IR" and the derived laplacian operator 2 = F V R + IR Under such discretization scheme, solving for the indicator function amounts to solving the linear system by
for d 3. We have the indicator function X E R",
inverting the divergence operator subject to boundary conditions of surface points having zero level set. Following [28], we fix the overall scale to m 0.5 at x = 0:
x -v.v S.t.
X|xe(e] = 0 and abs(x(x=0) = m
(2)
Point Rasterization: We obtain the uniformly discretized point normal field by rasterizing the point normals onto a uniformly sampled voxel grid. We can differentiably perform point rasterization via inverse trilinear interpolation, similar the approach in [28, 29]. We scatter the point normal values to the voxel grid vertices, weighted by the trilinear interpolation weights. The point rasterization process has ((n) space complexity, linear with respect to the number of grid cells, and ((N) time complexity, linear with respect to the number of points. See supplementary for details.
Spectral Methods for Solving PSR: In contrast to the finite-element approach taken in [28, we solve the PDEs using spectral methods [7]. While spectral methods are commonly used in scientific computing for solving PDEs and in some cases applied to computer vision problems [32]. we are the first to apply them in the context of Poisson Surface Reconstruction. Unlike finite-element approaches that depend on irregular data structures such as octrees or tetrahedral meshes for discritizing space, spectral methods can efficently solved over a uniform grid as they leverage highly optimized Fast Fourier Transform (FFT) operations that are well supported for GPUs, TPUs, and mainstream deep learning frameworks. Spectral methods decompose the original signal into a linear sum functions represented using the sine cosine basis functions whose derivatives This allows us to easily approximate differential operators in spectral space. We denote the spectral domain signals with a tilde symbol, i.e v FFT(v). We first solve for the unnormalized indicator function X" not accounting for boundary conditions
be computed analytically.
iu V
(n. = exp (-2 p2
X' = (FF
x= Do. (u) @ 2m 1
(3)
where the spectral frequencies are denoted as u == (u, v. w) E Rxd corresponding to the x, y, spatial dimensions, and IFFT(X) represents the inverse fast Fourier transform of X. Zo. (u) is Gaussian smoothing kernel of bandwidth a at grid resolution in the spectral domain. The Gaussian kernel is used mitigate ringing effects the point normals. We denote the element-wise product as @ : R" x R" + R", the no-norm as ||- 12 Raxd + R", and the dot product as (.) Rxxd Raxd + R". Finally, subtract by the mean of the indicator function at the point set and scale the indicator function to obtain the solution to the PSR problem in Eqn. 2:
result of the Gibbs phenomenon from rasterizing
abs abs(x'Ix=n) (x'
m
1
Σ X
KcH DE (c)
scale
subtraci by
detailed derivation our differentiable PSR solver is provided in the supplementary material.
3.2 SAP for Optimization-based 3D Reconstruction
We can the proposed differentiable Poisson solver for various applications. First, consider the classical task of surface reconstruction from unoriented point clouds. The overall pipeline for this setting is illustrated in Fig. (top). We now provide details about each component.
Forward pass: It is natural to initialize the oriented 3D point cloud serving as 3D shape representa- tion using the noisy 3D input points and corresponding (estimated) normals. However, to demonstrate the flexibility and robustness of our model, we purposefully initialize our model using generic 3D sphere with radius r in our experiments. Given the orientated point cloud, we apply our Poisson solver obtain an indicator function grid, which can be converted a mesh using Marching Cubes [36].
Backward pass: For every point Pmnssh sampled from the mesh M, we calculate bi-directional 1.2 Chamfer Distance Lcp with respect to the input point cloud. To backpropagate the loss LcD through

---

Shape-As-
POREe Noraal
Optimization-base based Setting
DPSR
Marching Cubes
CD
Target
Features
DPSR
Marching Cubes
Offsets fo Normals So
Mesh Output
DPSR
Noisy Input
PSR
Sample
Indicator Function
Learning-baat based Setting
Ground Truth
Model Overview. Top: Pipeline for optimization basedd single object reconstruction. The Chamfer loss on the target point cloud backpropagated to the source point cloud w/ normals for optimization. Bottom:
Figure 1:
Pipeline for fearning-based surface reconstruction. Unlike the optimization- based setting, here we provide supervision at the indicator grid level, since we assume access to wa- tertight meshes for supervision, is common practice in learning-baa based single object reconstruction.
Pmosh point p in our source oriented point cloud, we decompose the gradient using the chain rule:
dCCD OCCD @pmesh πx
Öp Öpmesh
0x Op
All terms in (5) are differentialable except for the middle one (Panch which involves Marching Cubes.
0x
However, this gradient can be effectively approximated by the inverse surface normal [56]:
Pmeshh 0x
Dmnsh
where Dmesh the normal of the point Pmesh- Different from MeshSDF [56] that the gradients update the latent code of a pretrained implicit shape representation, our method updates the source point cloud using the proposed differentiable Poisson solver.
Resampling: increase the robustness the optimization process, we uniformly resample points and normals from the largest mesh component every 200 iterations, and replace all points in the original point clouds with the resampled ones. This resampling strategy eliminates outlier points that drift away during the optimization, and enforces more uniform distribution of points. We provide an ablation study supplementary.
To further decrease run-time, we consider coarse-to-fine strategy during optimiza- tion. More specifically, start optimizing from which we obtain a coarse shape. Next, we sample from this coarse mesh and continue opti- mization at resolution of 64 for 1000 iterations. We repeat this process until we reach the target resolution (2569) at which we acquire the final output mesh. See also supplementary.
Coarse-to-fine:
indicator grid resolution of 32 for 1000 iterations,
3.3 SAP for Learning-based 3D Reconstruction
We now consider the learning- based 3D reconstruction setting in which we train a conditional model that takes a noisy. unoriented point cloud as input and outputs 3D shape. More specifically, we train model to predict clean oriented point cloud, from which we obtain a watertight mesh using our Poisson solver and Marching Cubes. We leverage the differentiability of our Poisson solver to learn the parameters of this conditional model. Following common practice, we assume watertight meshes ground truth and consequently supervise directly with the ground truth indicator grid obtained
5

---

from these meshes. Fig. 1 (bottom) illustrates the pipeline of our architecture for the learning caaning-based surface reconstruction task.
Architecture: We first encode the unoriented input point cloud coordinates {c} into a feature ø The resulting feature should encapsulate both local and global information about the input point cloud. We utilize the convolutional point encoder proposed in [52] for this purpose, Note that in the following, will use (c) to denote the features at point c, dropping the dependency of & on the remaining points {c} for clarity. Also, we use 9 to refer to network parameters in general.
Given their features, we aim to estimate both offsets and normals for every input point in the point cloud {c}. We use a shallow Multi-Layer Perceptron (MLP) fa to predict the offset for c:
Ac fc (c)
where <(c) is obtained from the feature volume using trilinear interpolation. We predict & offsets per input point, where lo 2 1. We add the offsets Ac to the input point position and call the updated point position è. Additional offsets allow us to densify the point cloud, leading to enhanced reconstruction quality. We choose & 7 for all learning- based reconstruction experiments (see ablation study in Table 4). For each updated point é, we use a second ML.P go to predict its normal:
ñ go go de (c))
We use the same decoder architecture as in [52] for both fo and go. The network comprises 5 layers of ResNet blocks with a hidden dimension of 32. These two networks fa and ge do not share weights.
Training and Inference: During training, we obtain the estimated indicator grid X from the predicted point clouds (é, in) using our differentiable Poisson solver. Since we assume watertight and noise-free meshes for supervision, we acquire the ground truth indicator grid by running PSR on densely sampled point clouds of the ground truth meshes with the corresponding ground truth normals. This avoids running Marching Cubes at every iteration and accelerates training. We use the Mean Square Error (MSE) loss on the predicted and ground truth indicator grid:
LDSSR = 1|x - xI1
We implement all models in PyTorch [51] and use the Adam optimizer [30] with a learning rate of 5e-4. During inference, we use our trained model to predict normals and offsets, use DPSR to solve for the indicator grid, and run Marching Cubes [36] to extract meshes.
Experiments
Following the exposition in the previous section, we conduct two types of experiments to evaluate our method. First, we perform single object reconstruction from unoriented point clouds. Next, we apply our method to learning- based surface reconstruction on ShapeNet [9], using noisy point clouds with or without outliers as inputs.
Datasets: We use the following datasets for optimization based reconstruction: 1) Thingi [71], 2) Surface reconstruction benchmark (SRB) [65] and 3) D-FAUST [6]. Similar to prior works, use 5 objects per dataset [19, 23, 65]. For learning-baad based object-level reconstruction, we consider all 13 classes of the ShapeNet [9] subset, using the train/val/test split from [[1].
Baselines: In the optimization-basee reconstruction setting, we compare to network-based methods IGR [19] and Point2Mesh [23], as well as Screened Poisson Surface Reconstruction' (SPSR) [29] on plane-fitted normals. To ensure that the predicted normals are consistently oriented for SPSR, propagate the normal orientation using the minimum spanning tree [72]. For learning- based surface reconstruction. we compare against point-based Point Set Generation Networks (PSGN) [17], patch- based AtlasNet [20], voxel-based 3D-R2N2 [11]. and ConvONe [52], which has recently reported ofate-of-the-art results on this task. We use ConvOnet in their best-performing setting (3-plane encoders). SPSR is also used as baseline. In addition, to evaluate the importance of our differentiable PSR optimization, we design another point-based baseline. This baseline uses the same network architecture to predict points and normals. However, instead of passing them to our Poisson solver and
We use the official implementation http://giinbb.co///naaaaaaaaa///

---

Thingi10K
Thingi10K
SRB
D-FAUST
Input
IGR [19]
Point2Mesh [23]
SPSR [29]
Ours
GT mesh
Figure 2: Optimization-based 3D Reconstruction. Input point clouds are downsampled for visual- ization. Note that the ground truth of SRB is provided as point clouds.
Dataset
Method
Chamfer-Ly
F-Score (f)
Normal C. (t)
Time (8)
IGR [19]
0.440
0.505
0.692
1842.3
Point2Mesh [23]
0.109
0.656
0.806
3714.7
Thingi10K
SPSR [29]
0.223
0.787
0.896
9.3
Ours
0,054
0.940
0.947
370.1
IGR [19]
0.178
0.755
1847.6
Point2Mesh [23]
0.116
0.648
4707.9
SRB
SPSR [29]
0.232
0.735
9.2
Ours
0.076
0.830
326.0
[GR [19]
0.235
0.805
0.911
1857.2
Point2Mesh [23]
3678.7
D-FAUST
SPSR [29]
0.044
0.966
0.965
4.3
Ours
0.043
0.966
0.959
379.9
Table Optimization- 3D Reconstruction. Quantitative comparison datasets. Normal Consistency cannot be evaluated on SRB as this dataset provides only unoriented point clouds. Optimization time evaluated on a single GTX 1080Ti GPU for IGR, Point2Mesh and our method.
calculate CDPSR on the indicator grid, we directly supervise the point positions with bi-directional Chamfer distance, and an LI Loss on the normals as done in [37]. During inference, we also feed the predicted points and normals our PSR solver and run Marching Cubes to obtain meshes.
Metrics: We consider Chamfer Distance, Normal Consistency and F-Score with the default threshold of 1% for evaluation, and also report optimization & inference time.
4.1 Optimization-basee 3D Reconstruction
In this part, we investigate whether our method can be used for the single-object surface reconstruction task from unoriented point clouds scans. We consider three different types of 3D imputs: point clouds sampled from synthetic meshes [71] with Gaussian noise, real-world scans [65], and high- resolution scans of humans with comparably little noise [6].
Fig. 2 and Table 2 show that our method achieves superior performance compared to both classical methods and network- based approaches. Note that the objects considered in this task are challenging

---

Low Noise
High Noise
Outliess
Input
SPSR [29] 3D-R2N2 [11] AtlasNet [20] ConvONet [52]
Ours
GT mesh
Figure 3: 3D Reconstruction from Point Clouds on ShapeNet. Comparison of SAP to baselines
on 3 different setups. More results can be found in supplementary.
(a)
Noise=0.0005
(b)
Noise=0.0225
(c) Noise= Outliers=50%
Chamfer- L, F-Score Normal C. Chamffer-,, L, F-Score Normal C. Chamfer-L, F-Score Normal C. Runtime
SPSR [29]
0.298
0.612
0.772
0.499
0.324
0,604
1.317
0.164
0,636
PSGN [17]
0.147
0.259
0.151
0.247
0.736
0.007
0.010
3D-R2N2 [1]]
0.710
0.709
0.015
AtlasNet [20]
0.093
0.708
0.855
0.117
0.527
0.821
1.822
0.057
0.609
0.025
ConvONet [52]
0.044
0.942
0.938
0.066
0.849
0.913
0.052
0.916
0.929
0.327
Ours (wlo Epess)
0.044
0,942
0.935
0.067
0.841
0.907
0.085
0.819
0.903
0.064
Ours
0.034
0.975
0.944
0.054
0.896
0.917
0.038
0.959
0.936
0.064
Table 3: 3D Reconstruction from Point Clouds ShapeNet. Quantitative comparison between our method and baselines on the ShapeNet dataset (mean over 13 classes).
due to their complex geometry, thin structures, noisy and incomplete observations. While some of the baseline methods fail completely on these challenging objects, method achieves robust performance across all datasets.
In particular, Fig. 2 shows that IGR occasionally creates meshes in free space, as this is not penalized by its optimization objective when point clouds are unoriented. Both, Point2Mesh and our method alleviate this problem by optimizing for the Chamfer distance between the estimated mesh and the input point clouds. However, Point2Mesh requires an initial mesh as input of which the topology cannot be changed during optimization. Thus, relies SPSR to provide an initial mesh for objects with genus larger than 0 and suffers from inaccurate initialization [23]. Furthermore, compared to both IGR and Point2Mesh,
method converges faster.
While SPSR even more efficient, suffers from incorrect normal estimation on noisy input point clouds, which is non-trivial task on its own. In contrast, our method demonstrates more robust behavior as we optimize points and normals guided by the Chamfer distance. Note that in this single object reconstruction task, method is not able to complete large unobserved regions (c.g., the bottom of the person's feet in Fig. is unobserved and hence not completed). This limitation can be addressed using learning-bass based object-level reconstruction as discussed next.
To analyze whether our proposed differentiable Poisson solver is also beneficial for learning- based reconstruction, we evaluate our method on the single object reconstruction task using noise and outlier- augmented point clouds from ShapeNet as input to our method. We investigate the performance for
three different noise levels: (a) Gaussian noise with zero mean and standard deviation 0.005, (b) Gaussian noise with zero mean and standard deviation 0.025, (c) 50% points have the same noise as in a) and the other 50% points are outliers uniformly sampled inside the unit cube.

---

Chamfer
F-Score
NormalC
128
256
Offset 1x
0.041
0.952
0.928
Enc.
Grid
MC
Total
Enc.
Gnd
MC
Total
Otfsct 3x
0,039
0.958
0.934
ConvONet
0.010
0.280
0.037
0.327
0.010
3.798
0.299
4.107
Offsct 3x
0.039
0,957
0,934
Ours
0.013
0,012
0.039
0.064
0.019
0.140
0.374
0.533
Offset 7x
0.038
0.959
0.936
Enc.
0.043
0.939
0.928
Enc.
0.038
0.959
0.936
Table 4: Ablation Study. Left: Runtime breakdown (encoding, grid evaluation, marching cubes) for ConvONet vs. ours in seconds. Right: Ablation over number of offsets and 2D vs. 3D encoders.
4.2 Learning-based Reconstruction on ShapeNet
Fig. 3 and Table show results. Compared to the baselines, method achieves similar better results on all three metrics. The results show that, in comparison to directly using Chamfer loss on point positions and L.1 loss on point normals, settings as it directly supervises the indicator grid which implicitly determines the surface through the Poisson equation. SPSR fails when the noise level high or when there are outliers in the input point cloud. We achieve significantly better performances than other representations such as point clouds, meshes, voxel grids and patches. Moreover, find that our method is robust to strong outliers. We refer to the supplementary for more detailed visualizations on how SAP handles outliers.
DPSR loss can produce better reconstructions in all
Table 3 also reports the runtime for setting (a) for all GPU-accelerated methods using single NVIDIA GTX 1080Ti GPU, averaged over all objects of the ShapeNet test set. The baselines [11, 20] demonstrate fast inference time but suffer in terms of reconstruction quality while the neural implicit model [52] attains high quality reconstructions but suffers from slow inference. In contrast, our method is able to produce competitive reconstruction results at reasonably fast inference time. In addition, since ConyONet and our method share a similar reconstruction pipeline, we provide more detailed breakdown of the runtime at resolution of 128 and 256 voxels in Table We use the default setup from ConvONet As we can see from Table 4, the difference in terms of point encoding and Marching Cubes is marginal, but we gain more than 20 speed-up over ConvONet in evaluating the indicator grid. In total, we are roughly 5x and 8x faster regarding the total inference time at resolution of 128 and 256 voxels, respectively.
4.3 A Ablaioon Study
In this section, we investigate different architecture choices in the context of learning-basee recon- struction. We conduct our ablation experiments on ShapeNet for the third setup (most challenging).
Number of Offsets: From Table 4 we notice that predicting more offsets per input point leads to better performance. This can be explained by the fact that with more points the object surface, geometric details can be better preserved.
Point Cloud Encoder:
Here we compare two different point encoder architectures proposed in [52]: 2D encoder using canonical planes a resolution 642 pixels and 3D encoder using a feature volume with a resolution of 32 voxels. We find that the 3D encoder works best in this setting and hypothesize that this is due to the representational alignment with the 3D indicator grid.
5 Conclusion
We introduce Shape-As-Pointe novel shape representation which lightweight, interpretable and produces watertight meshes efficiently. We demonstrate its effectiveness for the task of surface reconstruction from unoriented point clouds in both optimization- based and learning- based settings. Our method is currently limited to small scenes due to the cubic memory requirements with respect to the indicator grid resolution. We believe that processing scenes in sliding- window manner and apace-adaptive data structures (e.g., octrees) will enable extending our method to larger scenes. Point cloud-based methods are broadly used in real-world applications ranging from household robots to self-driving cars, and hence share the same societal opportunities and risks other learning- based 3D reconstruction techniques.
To be consistent, use the Marching Cubes implementation from [60] for both ConvONet and ours.

---

Acknowledgement: Andreas Geiger was supported by the ERC Starting Grant LEGO-3D (850533) and the DFG EXC number 2064/1 project number 390727645. The authors thank the Max Planck ETH Center for Learning Systems (CLS) for supporting Songyou Peng and the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Michael Niemeyer. This work was supported by an NVIDIA research gift. We thank Matthias Niessner, Thomas Funkhouser, Hugues Hopp, Yue Wang for helpful discussions in early stages of this project. We also thank Xu Chen, Christian Reiser, Rémi Pautrat for proofreading.
References
[1] M. Atzmon and Y. Lipman. Sal: Sign agnostic learning of shapes from raw data. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 2
[2] A. Badki, O. Gallo, Kautz, and P. Sen. Meshlet priors for 3d mesh reconstruction. In Proc.
IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2020. 2,
[3] Y. Ben-Shabat and S. Gould. Deepfit: 3d surface fitting via neural network weighted least squares. In Proc. of the European Conf. on Computer Vision (ECCV), 2020. 3
[4] Y. Ben-Shabat, M. Lindenbaum, and A. Fischer. Nesti-net: Normal estimation for unstructured
3d point clouds using convolutional neural networks. In Proc. IEFE Conf. Computer Vision and Pattern Recognition (CVPR), 2019.
[5] F. Bernardini, Mittleman, H. Rushmeier, C. Silva, and G. Taubin. The ball-pivoting algorithm for surface reconstruction. IEEE Trans. Visualization and Computer Graphics (VCG), 1999. 3 [6] F. Bogo, J. Romero, G. Pons-Moll, and M. Black. Dynamic FAUST: registering human bodies in motion. Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2017. 6, 7 [7] C. Canuto, M. Y. Hussaini, A. Quarteroni, and T. A. Zang. Spectral methods: fundamentals in single domains. Springer Science & Business Media, 2007. 4
[8] R. Chabra, J. E. Lenssen, E. 1lg, T. Schmidt, J. Straub, S. Lovegrove, and R. Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In Proc. of the European Conf. on Computer Vision (ECCV), 2020. 3
[9] A. X. Chang, T. A. Funkhouser, L. J. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. Shapenet: An information-rich 3d model
repository. arXiv.org, 1512. 2015. 6
[10] 7. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 1,3
[11] C. B. Choy, D. Xu, Gwak, K. Chen, and Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Proc. of the European Conf. on Computer Vision
(ECCV), 2016. 2, 6, 8, 9
[12] A. Dai, C. Diller, and M. Nießner. SG-NN: sparse generative neural networks for self-supervised scene completion of RGB-D scans. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
[13] A. Dai, C. Diller, and M. Nieβner. Sg-nn: Sparse generative neural networks for self-supervised scene completion of rgb-d scans. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR). 2020.
[14] A. Dai, D. Ritchie, M. Bokeloh, S. Reed, J. Sturm, and M. Nießner. Scancomplete: Large- scale scene completion and semantic segmentation for 3d scans. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. 3
[15] H. Edelsbrunner and E. Mükkee Three-dimensional alpha shapes. ACM Trans. on Gruphics,
1994. 3
[16] P. Erler, P. Guerrero, S. Ohrhallinger, M. Wimmer, and N. J. Mitra. Points2surf: Learning implicit surfaces from point cloud patches. In Proc. of the European Conf. on Computer Vision (ECCV), 2020. 3
[17) H. Fan, H. Su, and L. J. Guibas. A point set generation network for 3d object reconstruction from a single image. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),
2017. 2. 6, 8, 9
[18] K. Genova, F. Cole, A. Sud, A. Sarna, and T. A. Funkhouser. Local deep implicit functions for
3d shape. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020, 3 [19] A. Gropp, I.. Yariv, N. Haim, M. Atzmon, and Y. Lipman. Implicit geometric regularization for learning shapes. In Proc. of the International Conf. on Machine learning (ICML), 2020. 3, 6, 7
10

---

[20] T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, and M. Aubry. AtlasNet: A papier-máche approach to learning 3d surface generation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. 2, 6, 8,
[21] P. Guerrero, Y. Kleiman, M. Ovsjanikov, and N. Mitra. Pepnet learning local shape properties
from raw point clouds. In Computer Graphics Forum, 2018. 3
[22] K. Gupta and M. Chandraker. Neural mesh flow: 3d manifold mesh generation via diffeomorphic
flows. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 2
[23] R. Hanocka, G. Metzer, R. Giryes, and D. Cohen-Or. Point2mesh: a self-prior for deformable
meshes. In ACM Trans, on Graphics, 2020. 3, 6, 7,8
[24] C. Jiang, J. Huang, A. Tagliasacchi, and L. J. Guibas. Shapeflow: Learnable deformation flows
among 3d shapes. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 2 [25] C. Jiang, P. Marcus, et al. Hierarchical detail enhancing mesh- shape generation with 3d generative adversarial network. arXiv preprint arXiv:1709.07555
2017. 3
[26] C. Jiang, A. Sud, A. Makadia, J. Huang, M. Nieliner, and T. Funkhouser. Local implicit grid representations for 3d scenes. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition
(CVPR), 2020. 1,
[27] C. M. Jiang, J. Huang, K. Kashinath, Prabhat, P. Marcus, and M. Nießner. Spherical cnns on unstructured grids. In Proc. of the International Conf. on Learning Representations (ICLR),
2019. 2
[28] M. M. Kazhdan, M. Bolitho, and H. Hoppe. Poisson surface reconstruction. In Proceedings of the Fourth Earographics Symposium on Geometry Processing, 2006. 3, 4
[29] M. M. Kazhdan and H. Hoppe. Screened poisson surface reconstruction. ACM Trans. on Graphics, 32(3):29, 2013. 3, 4, 6, 7,8
[30] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proc. of the International Conf. on Machine learning (ICML), 2015. 6
[31] J. E. Lenssen, C. Osendorfer, and Masei. Deep iterative surface normal estimation. In Proc.
IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2020. 3
[32] J. Li and A. O. Hero. A spectral method for solving elliptic equations for surface reconstruction
and 3d active contours. In Proc. IEEE International Conf. on Image Processing (ICIP), 2001.
[33] Y. LLiao, S. Donne, and A. Geiger. Deep marching cubes: Learning explicit surface representa- tions. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. 3 [34] S. Lionar, D. Emtsev, D. Svilarkovic, and S. Peng. Dynamic plane convolutional occupancy networks. In Proc. of the IEEE Winter Conference on Applications of Computer Vision ( (WACV),
2021.
[35] S. Liu, Y. Zhang, S. Peng. B. Shi, M. Pollefeys, and Z. Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 3
[36] W. F. Lorensen and H. F. Cline. Marching cubes: A high resolution 3d surface construction
algorithm. In ACM Trans. on Graphics, 1987. 4, 6
[37] Q Ma, S. Saito, J. Yang, S. Tang, and M. J. Black. Scale: Modeling clothed humans with a surface codec of articulated local elements. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. 7,7
[38] Q. Ma, J. Yang. S. Tang, and M. J. Black. The power of points for modeling humans in clothing.
In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021. 2
[39] R. Martin-Brualla N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. 3
[40] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 1, 2,
[41] M. Meshry, D. B. Goldman, S. Khamis, H. Hoppe. R. Pandey, N. Snavely, and R. Martin- Brualla. Neural rerendering in the wild. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition ( (CVPR), 2019. 1
[42] G. Metzer, R. Hanocka, D. Zorin, R. Giryes, D. Panozzo, and D. Cohen-Or. Orienting point clouds with dipole propagation. ACM Trans. on Graphics, 2021.
[43] M. Mihajlovic, S. Weder, M. Pollefeys, and M. R. Oswald. Deepsurfels: Learning online appearance fusion. In Proc. IEFE Conf. on Computer Vision and Puttern Recognition (CVPR).
2021. 2
11

---

[44] M. Mihajlovic, Y. Zhang, M. J. Black, and S. Tang. LEAP: Learning articulated occupancy of people. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. [45] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proc. of the European Conf.
on Computer Vision (ECCV), 2020. 1,
[46] M. Niemeyer and A. Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2021.
[47] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. IEEE Conf. on Computer
Vision and Puttern Recognition (CVPR), 2020. 1,
[48] M. Oechsle, L. Mescheder,
M. Niemeyer, T. Strauss, and A. Geiger. Texture fields: Learning texture representations in function space. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
[49] M. Oechsle, S. Peng, and A. Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021.
[50] J. J. Park, P. Florence, J. Straub, R. A. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2019. 1,
[51] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. De M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems
(NeurIPS), 2019. 6
[52] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger. Convolutional occupancy networks. In Proc. the European Conf. on Computer Vision (ECCV), 2020. 3, 6, 8, 9 [53] C. R. Qi, H. Su, K. Mo, and I.. Guibas. Pointnet: Deep learning on point sets for 3d classifi- cation and segmentation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition
(CVPR), 2017. 2
[54] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In Advances in Neural Information Processing Systems (NeurlPS). 2017. [55] C. Reiser, S. Peng, Y. Liao, and A. Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mips. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021.
[56] E. Remelli, A. Lukoianov, S. R. Richter, B. Guillard, T. Bagautdinov, P. Baque, and P. Fua. Meshsdf: Differentiable iso-surface extraction. In Advances in Neural Information Processing
Systems (NeurIPS), 2020. 5
[57] V. Sitzmann. J. N. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing
Systems (NeurIPS), 2020. 3
[58] V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR). 2019. 1
[59] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich- Keil, N. Raghavan, U. Singhal, R. Ra- mamoorthi, J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems (NeurIPS),
2020. 3
[60] S. Van der Walt, J. L. Schönberger, J. Nunez-lglesias, F. Boulogne, J. D. Warner, N. Yager, E. Gouillart, and T. Yu. scikit-image: image processing in python. PeerJ, 2014. 9
[61] N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proc. of the European Conf. on Computer Vision (ECCV),
2018.
[62] S. Wang, M. Mihajlovic, Q. Ma, A. Geiger, and S. Tang. Metaavatar: Learning animatable clothed human models from few depth images. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 3
[63] W. Wang, Q. Xu, D. Ceylan, R. Mech, and U. Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 3
12

---

[64] Y. Wang, Y. Sun. Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph enn for learning on point clouds. ACM Trans. on Graphics, 2019. [65] Williams, T. Schneider, C. Silva, D. Zorin, J. Bruna, and D. Panozzo. Deep geometric prior for surface reconstruction. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 2, 3, 7
[66] J. Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2016. 3
[67] G. Yang, X. Huang. Z. Hao, M. Liu, S. Belongie, and B. Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proc. of the IEEE International Conf. on
Computer Vision (ICCV), 2019. 2
[68] Y. Yang, C. Feng, Y. Shen, and D. Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. [69] Z. Yang, Y. Chai, D. Anguelov, Y. Zhou, P. Sun, D. Erhan, S. Rafferty, and H. Kretzschmar. Surfelgan: Synthesizing realistic sensor data for autonomous driving. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2020. 2
[70] L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, B. Ronen, and Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[71] Q Zhou and A. Jacobson. Thingi10k: A dataset of 10,000 3d- ao-pprinting models. arXiv preprint Xrxiv: 1605. 04797, 2016. 6,7
[72] Q--. Zhou, Park, and V. Koltun. Open3D: modern library for 3D data processing. MrXiv::1000 09847700009900.000 2018. 6
13


Reviews:
[{"review_id": "UNsDT38ZYQI", "reviewer": "Reviewer_QqU1", "summary": "This work proposes a novel shape representation using point clouds. This is enabled by unique differentiable poisson solver layer that enables to convert the oriented point clouds to full volume representing the shape. I think the main contribution of this work is the spectral technique that solves poisson surface reconstruction problem, which can be efficiently integrated into neural workflows. While this work mostly focuses on surface reconstruction, I believe this representation can become broadly applicable to various problem domains.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has both high-level novelty (\"shape-as-points\" representation) as well as low-level technical contribution (spectral differentiable poisson solver). I think the proposed representation is valuable and will inspire future work. The evaluation is thorough and the method offers several quantitative advantages over state of the art methods (in terms of runtime and quality). I would be excited to see this work at NeurIPS. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are adequately addressed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "Mo43dnSacNa", "reviewer": "Reviewer_nyEb", "summary": "This paper proposes a differentiable adaptation of the Poisson Surface Reconstruction algorithm for oriented point sets. It discretizes the Poisson equation over a 3D grid and use spectral methods to find the solution for indicator function values over this 3D grid. The final grid is fed to marching cubes algorithm to obtain the final mesh. The differentiability of the approach allows to use the proposed method in various optimization-based applications from which the authors consider optimization-based and learning-based 3D reconstruction tasks. Proposed models are evaluated on datasets with different level of details and noise present in the inputs and compared to both classical and more recent learning-based methods.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* The paper is well-written.\n* The proposed approach is novel and promising, both qualitative and quantitative results shown are strong.\n* Evaluations are extensive and include both different datasets and different noise injection configurations. The authors additionally provide evaluations of the efficiency of the proposed method as well as some additional ablation studies.\n\nCons:\n* The choice of some baselines for learning-based approaches is questionable. None of PSGN, 3D-R2N2 and AtlasNet models were designed for surface reconstruction from input unoriented points, so they were not optimized for this task. There is a least one very relevant work left unmentioned and uncompared to in this submission [1].\n* It is clear that it is hard to provide a full mathematical explanation in the main text without using too much space but even in the supplementary materials some detail are missing. My concern is mainly equation 7 from the supplementary. Equations 5 and 6 partly justify it but do not account for the appearance of $\\tilde{g}$. It is said that it is introduced to mitigate Gibbs phenomenon, but it is not obvious why the addition of $\\tilde{g}$ to the solution of Poisson equation in frequency domain do not corrupt this solution (there will not be an identity if one put $\\tilde{\\chi}$ in FFT of Poisson equation).\n\nOverall, I believe this is a strong submission. The cons should be addressed but, in fact, are minor in my opinion and should not justify rejection.\n\nMinor comments:\n1) L146-148 \"$\\tilde{g}_{\\sigma, r}(u)$...\" - rephrase the sentence, something is missing.\n2) L183 \"a a\"\n\n[1] Erler, P., Guerrero, P., Ohrhallinger, S., Mitra, N. J., Wimmer, M. Points2surf: learning implicit surfaces from point clouds. In ECCV’20.\n\n***\nAfter reading the rebuttal and other reviews I decided to improve my rating since my concerns were answered. This is an overall strong submission with great novelty and promising experimental results despite the limitations of cubic complexity for indicator grids.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not provide any discussions about these impacts besides stating that their approach shares the risks with all the other learning-based 3D reconstruction approaches.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "J5syVm6nGfm", "reviewer": "Reviewer_eo1g", "summary": "This work introduces a differentiable points (+normals) to mesh layer based on a differentiable formulation of Poisson surface reconstruction.\nFirst, the proposed method is validated in point cloud optimisation tasks where the objective function is expressed w.r.t surface meshes.\nThen, the method is proposed as a way to represent 3D shapes with oriented point cloud in a learning setting.\nCrucially, this results in a shape parameterization, dubbed Shape As Points (SAP), that is more lightweight and faster than neural implicit fields, while still yielding watertight shapes and being able to handle arbitrary topology. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- GPU-accelerated differentiable pipeline to solve Poisson equation could be useful to research community at large \n\n- New shape parameterisation has similar properties to neural implicit fields (can handle arbitrary topology, produces watertight surfaces) while being more lightweight and efficient\n\n- Extensive experimental evaluation demonstrate the advantages of the proposed representation when 3D supervision is available \n\nWeaknesses:\n\n- Limited resolution: the proposed pipeline is, by design, limited by the cubic memory requirement at training time. This is not the case for neural implicit fields, and in some sense the proposed representation is a step-back from them, which allowed to represent arbitrary topology while breaking free from memory intensive 3D grids.\n\n- 2D supervision: the proposed approach is not evaluated in a context in which 3D supervision is not available, although, in theory, any off the shelf differentiable renderer would be compatible with the pipeline. Without this evaluation, it is hard to say if the proposed approach has any fundamental limitation in a differentiable rendering context (e.g. due to its underlying sparsity?).\n\nQuestion for the authors:\n\nalong lines 45~48 authors claim: \"It is easy to initialize SAP with a given geometry such as template shapes or noisy observations. In contrast, neural implicit representations are harder to initialize, except for few simple primitives like spheres\". The claim is re-stated in table 1. I don't understand what do authors exactly mean with this: \nIn the learning setting, where oriented clouds of points are used as underlying 3D shape representation, we do not really care about initialisation. The same though holds for neural implicit fields, as the weights of the underlying MLP are similarly initialised using standard techniques. \nIn the optimization setting, although I understand that the SAP framework would allow for initialising the shape using noisy points and normals, the reported experiments still decide to start from a sphere (see supplemental video), probably because this results in a better local minima attained by the optimisation process. Note that a sphere is indeed a trivial initialisation for  an implicit field, as correctly stated by the authors along lines 45~48. \nIn light of all this, I think this claim should be either better argumented/explained or toned down.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As stated above, I think the cubic memory requirement at training time should have been better pointed out in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "8Vsp1Igricg", "reviewer": "Reviewer_KwTG", "summary": "The paper formulates a differentiable Poisson Surface Reconstruction layer, which maps an oriented point cloud to an indicator function over the interior of the object. The Poisson solver is done in the spectral domain on the GPU for computational efficiency. It shares many of its advantages with other neural implicit representations, but because it produces an indicator function over the whole space, there is no need to query a network over an entire voxel grid. \n\nThe layer is incorporated into two tasks: an optimization-based 3D shape reconstruction that fits a spherical point cloud initialization to a target point cloud, and a learning-based 3D reconstruction task that matches noisy point cloud input to a ground truth mesh. The non-differentiable step of Marching Cubes in these pipelines has its gradient approximated with the inverse surface normal, which is quite natural. The experimental results seem good, and beat out the existing state-of-the-art, and an ablation study is performed as well on the learning-based task to investigate the form of the feature extraction network there.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the method is an interesting and novel implementation of the PSR algorithm, used to produce a better implicit representation than the existing state-of-the-art methods. I'm generally convinced by the experimental results, though I would appreciate some more information on the structure of the offset and normal prediction network. It was not quite clear to me why the particular architecture choices were made and whether other options were considered. The ablation study sticks to the convolutional point cloud encoder that they noted earlier, and considers two different versions, but does not consider other network architectures.\n\nWith respect to the layer, I was also curious to know about how much spectral resolution was used in the Poisson solver, and how the sigma was chosen for the Gaussian smoothing kernel.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors did a fair job of addressing these points.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Congratulations, the paper is accepted to NeurIPS 2021!
Please incorporate additional experiments, edits and corrections as discussed in rebuttal. 
Decision: Accept (Oral)