Shape As Points: A Differentiable Poisson Solver

1 Introduction . Shape representations are central to many of the recent advancements in 3D computer vision and computer graphics , ranging from neural rendering [ 41,45,48,55,58 ] to shape reconstruction [ 10,26,40 , 47 , 50 , 52 , 70 ] . While conventional representations such as point clouds and meshes are efficient and well-studied , they also suffer from several limitations : Point clouds are lightweight and easy to obtain , but do not directly encode surface information . Meshes , on the other hand , are usually restricted to fixed topologies . More recently , neural implicit representations [ 10 , 40 , 50 ] have shown promising results for representing geometry due to their flexibility in encoding varied topologies , and their easy integration with differentiable frameworks . However , as such representations implicitly encode surface information , extracting the underlying surface is typically slow as they require numerous network evaluations in 3D space for extracting complete surfaces using marching cubes [ 10 , 40 , 50 ] , or along rays for intersection detection in the context of volumetric rendering [ 45 , 47 , 49 , 70 ] . In this work , we introduce a novel Poisson solver which performs fast GPU-accelerated Differentiable Poisson Surface Reconstruction ( DPSR ) and solves for an indicator function from an oriented point cloud in a few milliseconds . Thanks to the differentiablility of our Poisson solver , gradients from a loss on the output mesh or a loss on the intermediate indicator grid can be efficiently backpropagated to update the oriented point cloud representation . This differential bridge between points , indicator ∗Work done while at UC Berkeley . †Corresponding authors . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . functions , and meshes allows us to represent shapes as oriented point clouds . We therefore call this shape representation Shape-As-Points ( SAP ) . Compared to existing shape representations , Shape-AsPoints has the following advantages ( see also Table 1 ) : Efficiency : SAP has a low memory footprint as it only requires storing a collection of oriented point samples at the surface , rather than volumetric quantities ( voxels ) or a large number of network parameters for neural implicit representations . Using spectral methods , the indicator field can be computed efficiently ( 12 ms at 1283 resolution3 ) , compared to the typical rather slow query time of neural implicit networks ( 330 ms using [ 40 ] at the same resolution ) . Accuracy : The resulting mesh can be generated at high resolutions , is guaranteed to be watertight , free from self-intersections and also topology-agnostic . Initialization : It is easy to initialize SAP with a given geometry such as template shapes or noisy observations . In contrast , neural implicit representations are harder to initialize , except for few simple primitives like spheres [ 1 ] . See supplementary for more discussions . To investigate the aforementioned properties , we perform a set of controlled experiments . Moreover , we demonstrate state-of-the-art performance in reconstructing surface geometry from unoriented point clouds in two settings : an optimization-based setting that does not require training and is applicable to a wide range of shapes , and a learning-based setting for conditional shape reconstruction that is robust to noisy point clouds and outliers . In summary , the main contributions of this work are : • We present Shape-As-Points , a novel shape representation that is interpretable , lightweight , and yields high-quality watertight meshes at low inference times . • The core of the Shape-As-Points representation is a versatile , differentiable and generalizable Poisson solver that can be used for a range of applications . • We study various properties inherent to the Shape-As-Points representation , including inference time , sensitivity to initialization and topology-agnostic representation capacity . • We demonstrate state-of-the-art reconstruction results from noisy unoriented point clouds at a significantly reduced computational budget compared to existing methods . Code is available at https : //github.com/autonomousvision/shape_as_points . 2 Related Work . 2.1 3D Shape Representations . 3D shape representations are central to 3D computer vision and graphics . Shape representations can be generally categorized as being either explicit or implicit . Explicit shape representations and learning algorithms depending on such representations directly parameterize the surface of the geometry , either as a point cloud [ 17 , 38 , 53 , 54 , 64 , 67 ] , parameterized mesh [ 22 , 24 , 27 , 61 ] or surface patches [ 2 , 20 , 37 , 43 , 65 , 68 , 69 ] . Explicit representations are usually lightweight and require few parameters to represent the geometry , but suffer from discretization , the difficulty to represent 3On average , our method requires 12 ms for computing a 1283 indicator grid from 15K points on a single NVIDIA GTX 1080Ti GPU . Computing a 2563 indicator grid requires 140 ms. watertight surfaces ( point clouds , surface patches ) , or are restricted to a pre-defined topology ( mesh ) . Implicit representations , in contrast , represent the shape as a level set of a continuous function over a discretized voxel grid [ 14 , 25 , 33 , 66 ] or more recently parameterized as a neural network , typically referred to as neural implicit functions [ 10 , 40 , 50 ] . Neural implicit representations have been successfully used to represent geometries of objects [ 10 , 16 , 18 , 35 , 40 , 44 , 47 , 50 , 57 , 59 , 62 , 63 ] and scenes [ 8 , 26 , 34 , 46 , 52 , 57 ] . Additionally , neural implicit functions are able to represent radiance fields which allow for high-fidelity appearance and novel view synthesis [ 39,45 ] . However , extracting surface geometry from implicit representations typically requires dense evaluation of multi-layer perceptrons , either on a volumetric grid or along rays , resulting in slow inference time . In contrast , SAP efficiently solves the Poisson Equation during inference by representing the shape as an oriented point cloud . 2.2 Optimization-based 3D Reconstruction from Point Clouds . Several works have addressed the problem of inferring continuous surfaces from a point cloud . They tackle this task by utilizing basis functions , set properties of the points , or neural networks . Early works in shape reconstruction from point clouds utilize the convex hull or alpha shapes for reconstruction [ 15 ] . The ball pivoting algorithm [ 5 ] leverages the continuity property of spherical balls of a given radius . One of the most popular techniques , Poisson Surface Reconstruction ( PSR ) [ 28,29 ] , solves the Poisson Equation and inherits smoothness properties from the basis functions used in the Poisson Equation . However , PSR is sensitive to the normals of the input points which must be inferred using a separate preprocessing step . In contrast , our method does not require any normal estimation and is thus more robust to noise . More recent works take advantage of the continuous nature of neural networks as function approximators to fit surfaces to point sets [ 19 , 23 , 42 , 65 ] . However , these methods tend to be memory and computationally intensive , while our method yields high-quality watertight meshes in a few milliseconds . 2.3 Learning-based 3D Reconstruction from Point Clouds . Learning-based approaches exploit a training set of 3D shapes to infer the parameters of a reconstruction model . Some approaches focus on local data priors [ 2 , 26 ] which typically result in better generalization , but suffer when large surfaces must be completed . Other approaches learn objectlevel [ 33 , 40 , 50 ] or scene-level priors [ 12 , 13 , 26 , 52 ] . Most reconstruction approaches directly reconstruct a meshed surface geometry , though some works [ 3 , 4 , 21 , 31 ] first predict point set normals to subsequently reconstruct the geometry via PSR [ 28 , 29 ] . However , such methods fail to handle large levels of noise , since they are unable to move points or selectively ignore outliers . In contrast , our end-to-end approach is able to address this issue by either moving outlier points to the actual surface or by selectively muting outliers either by forming paired point clusters that self-cancel or reducing the magnitude of the predicted normals which controls their influence on the reconstruction . 3 Method . At the core of the Shape-As-Points representation is a differentiable Poisson solver , which can be used for both optimization-based and learning-based surface estimation . We first introduce the Poisson solver in Section 3.1 . Next , we investigate two applications using our solver : optimization-based 3D reconstruction ( Section 3.2 ) and learning-based 3D reconstruction ( Section 3.3 ) . 3.1 Differentiable Poisson Solver . The key step in Poisson Surface Reconstruction [ 28 , 29 ] involves solving the Poisson Equation . Let x ∈ R3 denote a spatial coordinate and n ∈ R3 denote its corresponding normal . The Poisson Equation arises from the insight that a set consisting of point coordinates and normals { p = ( c , n ) } can be viewed as samples of the gradient of the underlying implicit indicator function χ ( x ) that describes the solid geometry . We define the normal vector field as a superposition of pulse functions v ( x ) = ∑ ( ci , ni ) ∈ { p } δ ( x − ci , ni ) , where δ ( x , n ) = { n if x = 0 and 0 otherwise } . By applying the divergence operator , the variational problem transforms into the standard Poisson equation : ∇2χ : = ∇ · ∇χ = ∇ · v ( 1 ) In order to solve this set of linear Partial Differential Equations ( PDEs ) , we discretize the function values and differential operators . Without loss of generality , we assume that the normal vector field v and the indicator function χ are sampled at r uniformly spaced locations along each dimension . Denote the spatial dimensionality of the problem to be d. Without loss of generality , we consider the three dimensional case where n : = r × r × r for d = 3 . We have the indicator function χ ∈ Rn , the point normal field v ∈ Rn×d , the gradient operator ∇ : Rn 7→ Rn×d , the divergence operator ( ∇· ) : Rn×d 7→ Rn , and the derived laplacian operator ∇2 : = ∇ · ∇ : Rn 7→ Rn . Under such a discretization scheme , solving for the indicator function amounts to solving the linear system by inverting the divergence operator subject to boundary conditions of surface points having zero level set . Following [ 28 ] , we fix the overall scale to m = 0.5 at x = 0 : χ = ( ∇2 ) −1∇ · v s.t . χ|x∈ { c } = 0 and abs ( χ|x=0 ) = m ( 2 ) Point Rasterization : We obtain the uniformly discretized point normal field v by rasterizing the point normals onto a uniformly sampled voxel grid . We can differentiably perform point rasterization via inverse trilinear interpolation , similar to the approach in [ 28 , 29 ] . We scatter the point normal values to the voxel grid vertices , weighted by the trilinear interpolation weights . The point rasterization process has O ( n ) space complexity , linear with respect to the number of grid cells , and O ( N ) time complexity , linear with respect to the number of points . See supplementary for details . Spectral Methods for Solving PSR : In contrast to the finite-element approach taken in [ 28,29 ] , we solve the PDEs using spectral methods [ 7 ] . While spectral methods are commonly used in scientific computing for solving PDEs and in some cases applied to computer vision problems [ 32 ] , we are the first to apply them in the context of Poisson Surface Reconstruction . Unlike finite-element approaches that depend on irregular data structures such as octrees or tetrahedral meshes for discritizing space , spectral methods can be efficently solved over a uniform grid as they leverage highly optimized Fast Fourier Transform ( FFT ) operations that are well supported for GPUs , TPUs , and mainstream deep learning frameworks . Spectral methods decompose the original signal into a linear sum of functions represented using the sine / cosine basis functions whose derivatives can be computed analytically . This allows us to easily approximate differential operators in spectral space . We denote the spectral domain signals with a tilde symbol , i.e. , ṽ = FFT ( v ) . We first solve for the unnormalized indicator function χ′ , not accounting for boundary conditions χ′ = IFFT ( χ̃ ) χ̃ = g̃σ , r ( u ) iu · ṽ −2π‖u‖2 g̃σ , r ( u ) = exp ( − 2σ 2‖u‖2 r2 ) ( 3 ) where the spectral frequencies are denoted as u : = ( u , v , w ) ∈ Rn×d corresponding to the x , y , z spatial dimensions , and IFFT ( χ̃ ) represents the inverse fast Fourier transform of χ̃ . g̃σ , r ( u ) is a Gaussian smoothing kernel of bandwidth σ at grid resolution r in the spectral domain . The Gaussian kernel is used to mitigate ringing effects as a result of the Gibbs phenomenon from rasterizing the point normals . We denote the element-wise product as : Rn × Rn 7→ Rn , the L2-norm as ‖ · ‖2 : Rn×d 7→ Rn , and the dot product as ( · ) : Rn×d × Rn×d 7→ Rn . Finally , we subtract by the mean of the indicator function at the point set and scale the indicator function to obtain the solution to the PSR problem in Eqn . 2 : χ = m abs ( χ′|x=0 ) ︸ ︷︷ ︸ scale ( χ′ − 1 | { c } | ∑ c∈ { c } χ′|x=c ) ︸ ︷︷ ︸ subtract by mean ( 4 ) A detailed derivation of our differentiable PSR solver is provided in the supplementary material .

Summary: The paper formulates a differentiable Poisson Surface Reconstruction layer, which maps an oriented point cloud to an indicator function over the interior of the object. The Poisson solver is done in the spectral domain on the GPU for computational efficiency. It shares many of its advantages with other neural implicit representations, but because it produces an indicator function over the whole space, there is no need to query a network over an entire voxel grid.   The layer is incorporated into two tasks: an optimization-based 3D shape reconstruction that fits a spherical point cloud initialization to a target point cloud, and a learning-based 3D reconstruction task that matches noisy point cloud input to a ground truth mesh. The non-differentiable step of Marching Cubes in these pipelines has its gradient approximated with the inverse surface normal, which is quite natural. The experimental results seem good, and beat out the existing state-of-the-art, and an ablation study is performed as well on the learning-based task to investigate the form of the feature extraction network there.