Geometry Processing with Neural Fields

1 Introduction . In many graphics applications , users may want to edit digital shapes using just a few clicks , such as making a character bow by dragging the head downwards . Such manipulation from sparse input requires geometry processing algorithms . Most of these algorithms use polygonal meshes to represent shapes [ 11 ] . Polygonal meshes were created initially for researchers as a representation of real-world shapes that they can both interpret and manipulate [ 14 , 52 , 78 ] . However , developing algorithms that automatically manipulate meshes is often difficult since they involve discretizing the surface . For example , changing the topology of a mesh , such as turning a sphere into a torus , will break such discretization and require additional repair procedures such as remeshing [ 2 , 3 ] . Given that shape editing is increasingly performed by algorithms , it seems worthwhile to search for a shape representation that is more amenable to automatic geometry processing . An alternative to the polygonal mesh is an implicit representation , in which the surface is represented by a level set of a field : { x|f ( x ) = c } [ 22 , 39 , 63 ] . Since it is easy to change shape topology using implicit representations , people have applied them for geometry processing tasks such as shape merging [ 8 , 51 , 54 ] . These works store implicit fields using voxels or octrees , which introduce memory-intensive spatial discretization . Recent research addresses this by using continuous neural networks to represent implicit fields [ 19 , 45 , 46 , 56 ] . These neural fields possess several advantages in addition to the merits inherited from implicit representation : they are compact to store [ 23 , 45 ] and can produce high-quality continuous surfaces at arbitrary resolutions [ 45 , 56 ] . The community has achieved compelling results using neural fields in a variety of applications [ 47 , 64 , 73 , 84 ] . Notwithstanding these results , it is still unclear whether shape editing tasks challenging for implicit fields ( e.g. , deformation and filtering ) can be performed with neural fields . In this paper , we ask : can geometry processing be done entirely using neural fields ? ∗Email : gy46 @ cornell.edu . This work is done while Guandao was interning at Intel Labs . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Geometry processing tasks are challenging because they are typically under-constrained : user input is very sparse . Therefore , we need to leverage priors that characterize how a natural surface behaves . Such surface priors are usually instantiated by minimizing energy functionals that describe the physical properties of surfaces using operators from differential geometry . For example , one can encourage surfaces to be smooth by minimizing the curvature , which can be measured by the LaplaceBeltrami operator . Such operators , however , require a parameterization of the surface , which is not available for level sets of a neural field . Mesh algorithms usually approximate these geometric operators using the geodesic neighbors of the surface point . This allows the algorithms to focus on intrinsic geometric properties that are independent of shape parameterization . However , it is not easy to obtain geodesic neighbors in a neural field since the surface of interest is encoded implicitly by a set of points that evaluate the field function to the same value . We posit that computing these geometric operators in neural fields requires a fundamentally different approach . The idea is to approximate the local surface of the level set using the derivatives of the underlying field . We can evaluate intrinsic geometry properties of the level set , such as curvature , using only the field derivatives [ 39 , 54 ] . This allows us to develop loss functions that describe surface priors such as elasticity or rigidity . To achieve this , we use the fact that neural fields are designed to be infinitely differentiable [ 66 , 74 ] . The infinite differentiability of neural fields makes it possible to optimize loss functions that involve higher-order derivatives using gradient descent methods . Thus , unlike mesh-based geometry processing algorithms that approximate these objectives using surface discretizations , we can directly optimize in terms of the derivatives of the field . Our formulation provides a proof of concept that geometry processing can be done entirely using neural fields . In particular , we focus on two tasks : shape filtering ( e.g. , smoothing and sharpening ) and topology-preserving deformations . These two tasks are not only essential for downstream applications but also bring out the known challenges associated with implicit representations . We first tackle shape sharpening and smoothing and show how these can be done by optimizing a loss function based on geometric properties computed via the neural field ’ s derivatives . To achieve shape deformation , we propose to warp the neural field using a deformation vector field modeled by an invertible neural network [ 7 ] . With this invertible deformation field , we derive a training objective that models the implicitly represented surfaces as elastic shells . We compare our method with well-established mesh-based baselines [ 70 , 76 ] . Our method can match the quality of the mesh-based counterparts without discretizing the surface . We hope that our work can inspire future generations of geometry processing algorithms using neural fields . 2 Related Work . This paper builds on two bodies of work : polygon mesh geometry processing and neural field representations . We will focus on prior work on two geometry processing tasks : shape filtering and topology-preserving deformation . Please refer to Botsch et al . [ 11 ] for further reading . Mesh smoothing ( and sharpening ) . The goal of mesh smoothing is to remove high-frequency noise and produce a smooth surface . Noise removal can be achieved by Fourier analysis [ 76 , 79 ] or by modeling surface motion with a diffusion equation [ 26 ] . Smoothness can be promoted by minimizing energy functionals inspired by differential operators [ 50 , 72 , 82 ] . Sharpening is the inverse of smoothing . In this paper , we adopt the goal of smoothing from prior work [ 20 , 76 ] and design a corresponding differentiable objective for neural fields . Implicit field smoothing . Implicit representations can be smoothed by evolving the level set according to its curvature normal [ 49 , 54 ] . Many network architectures , such as ReLU MLP , bias toward smoothness when modeling an implicit field [ 61 , 66 , 74 ] . Regularization objectives and initialization schemes have been proposed to obtain smooth level sets from scanned surface points [ 5 , 31 ] . These works mostly use smoothness as a prior when training neural fields . Our work formulates an objective that allows smoothing or sharpening a neural field as desired . Mesh deformation . Mesh deformation is done by either deforming the shape ( i.e . vertices ) [ 9 , 69 ] or the space [ 6 , 36 , 41 , 48 , 65 ] . These algorithms usually draw inspiration from physics and differential geometry to develop energy functions that encourage natural deformation [ 10 , 15 , 24 , 70 , 71 , 77 ] . They also linearize such objectives so that they can be solved efficiently using linear solvers [ 9 ] . Recent research also applies deep learning to optimize for non-linear deformation losses [ 32 , 34 , 38 , 80 , 81 , 83 ] . Deforming a mesh , however , requires estimating the deformation objectives with spatial discretization and maintaining vertex connectivity in a way that preserves topology [ 42 ] . We deform neural fields with a continuous invertible field to circumvent these issues . Implicit field deformation . Prior works have studied the deformation of implicit fields for tasks like physics simulation [ 39 , 51 , 54 , 55 ] . These works use voxels or octrees to represent implicit fields , while our paper advocates for using neural fields for deformation . There is some past work on deforming neural fields to match a target shape [ 12 , 18 , 37 , 53 , 58 ] or image [ 43 , 57 , 60 ] without any correspondences between the two . In contrast , we perform such deformation with very few localized user-provided correspondences , but without any other information about the target shape . Our setting is very convenient for artists , but it requires the algorithm to provide a strong surface prior due to the lack of a densely specified target state . Other prior works on editing neural fields are trained on a dataset of shapes from a particular object category [ 25 , 29 , 30 , 33 , 85 ] . Since surface priors are specified implicitly by the dataset , it is unclear how to apply these methods to out-of-distribution shapes . In contrast , our paper requires no training dataset and instead enforces priors inspired by more general physical properties such as smoothness and elasticity . Remelli et al . [ 62 ] developed a way to differentiate iso-surface extraction , which allows deforming neural fields using objectives carried over from mesh-based algorithms . Our method avoids iso-surface extraction entirely and formulates differentiable deformation objectives directly on the implicit field . 3 Computing Surface Properties of Neural Fields . In this section , we will discuss how to compute surface properties using neural fields and establish notation for the following sections . Let ∂Ω be a surface enclosing the region Ω . A signed distance field ( SDF ) for surface ∂Ω is defined as d ( x ) = s ( x ) miny∈∂Ω ∥x− y∥ . The sign function s ( x ) evaluates to −1 if x ∈ Ω and 1 otherwise . We will assume that a neural field approximate an SDF whose zero-isosurface represents the surface of interest . For a neural field f , we denote its zero-isosurface using Mf . This section summarizes how to compute surface normals and curvatures from neural fields . This is well-understood and we summarize this here for convenience [ 39 , 54 , 59 ] . Surface normal . The surface normal is the vector perpendicular to the tangent plane and describes the local orientation of the surface . For an SDF , the outward oriented surface normal of its level set is the field gradient : nd ( x ) = ∇xd ( x ) . It can be shown that the SDF ’ s gradient norm is always one : ∥∇xd ( x ) ∥ = 1 . This property can be used to ensure that the field remains a valid SDF throughout any manipulation [ 4 , 31 ] . The closest point in ∂Ω from x can be found by x − d ( x ) nd ( x ) . This property can be used to sample points from the isosurface without creating a mesh [ 5 , 13 ] . Finally , since the tangent plane is perpendicular to the surface normal , we can project vectors onto the tangent plane by subtracting their projection onto the normal using the following projection matrix x : Pd ( x ) = I− nd ( x ) nd ( x ) T . This projection matrix allows us to characterize the tangent plane . This matrix is important for measuring how much a given deformation stretches the tangent direction [ 35 ] . Curvature . Intuitively , curvature describes how much a surface deviates from a plane . This can be captured by the total derivative of the surface normal , which is also known as the shape operator : SD ( x ) = Dnd ( x ) . For an SDF , the normal itself is given by the derivative of the field , thus the shape operator for an SDF is the Hessian of the field function : Hd ( x ) = D2d ( x ) . The shape operator can be used to capture many different notions of curvature , including mean curvature ( half the trace of SD ( x ) , denoted by κ̄ ) , Gaussian curvature ( the determinant of SD ( x ) ) , and principal curvatures ( the eigenvalues of SD ( x ) ) . We will use these to define objectives that smooth or sharpen the surface .

Summary: This paper proposes a method to perform some common operations from geometry processing on shapes represented as neural implicit fields (i.e., deep SDFs) rather than parameterized meshes. In particulars, by utilizing the fact that high order derivatives of NIFs can be computed automatically, the authors formulate several loss terms, which allow them to apply sharpening, smoothing, or deformation to an existing implicit field via a reoptimization. They show qualitative and qualitative results and comparisons to the mesh-based analogs of their algorithm and present a fairly comprehensive ablation study validating their design choices.