Geometry Processing with Neural Fields

1 Introduction . In many graphics applications , users may want to edit digital shapes using just a few clicks , such as making a character bow by dragging the head downwards . Such manipulation from sparse input requires geometry processing algorithms . Most of these algorithms use polygonal meshes to represent shapes [ 11 ] . Polygonal meshes were created initially for researchers as a representation of real-world shapes that they can both interpret and manipulate [ 14 , 52 , 78 ] . However , developing algorithms that automatically manipulate meshes is often difficult since they involve discretizing the surface . For example , changing the topology of a mesh , such as turning a sphere into a torus , will break such discretization and require additional repair procedures such as remeshing [ 2 , 3 ] . Given that shape editing is increasingly performed by algorithms , it seems worthwhile to search for a shape representation that is more amenable to automatic geometry processing . An alternative to the polygonal mesh is an implicit representation , in which the surface is represented by a level set of a field : { x|f ( x ) = c } [ 22 , 39 , 63 ] . Since it is easy to change shape topology using implicit representations , people have applied them for geometry processing tasks such as shape merging [ 8 , 51 , 54 ] . These works store implicit fields using voxels or octrees , which introduce memory-intensive spatial discretization . Recent research addresses this by using continuous neural networks to represent implicit fields [ 19 , 45 , 46 , 56 ] . These neural fields possess several advantages in addition to the merits inherited from implicit representation : they are compact to store [ 23 , 45 ] and can produce high-quality continuous surfaces at arbitrary resolutions [ 45 , 56 ] . The community has achieved compelling results using neural fields in a variety of applications [ 47 , 64 , 73 , 84 ] . Notwithstanding these results , it is still unclear whether shape editing tasks challenging for implicit fields ( e.g. , deformation and filtering ) can be performed with neural fields . In this paper , we ask : can geometry processing be done entirely using neural fields ? ∗Email : gy46 @ cornell.edu . This work is done while Guandao was interning at Intel Labs . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Geometry processing tasks are challenging because they are typically under-constrained : user input is very sparse . Therefore , we need to leverage priors that characterize how a natural surface behaves . Such surface priors are usually instantiated by minimizing energy functionals that describe the physical properties of surfaces using operators from differential geometry . For example , one can encourage surfaces to be smooth by minimizing the curvature , which can be measured by the LaplaceBeltrami operator . Such operators , however , require a parameterization of the surface , which is not available for level sets of a neural field . Mesh algorithms usually approximate these geometric operators using the geodesic neighbors of the surface point . This allows the algorithms to focus on intrinsic geometric properties that are independent of shape parameterization . However , it is not easy to obtain geodesic neighbors in a neural field since the surface of interest is encoded implicitly by a set of points that evaluate the field function to the same value . We posit that computing these geometric operators in neural fields requires a fundamentally different approach . The idea is to approximate the local surface of the level set using the derivatives of the underlying field . We can evaluate intrinsic geometry properties of the level set , such as curvature , using only the field derivatives [ 39 , 54 ] . This allows us to develop loss functions that describe surface priors such as elasticity or rigidity . To achieve this , we use the fact that neural fields are designed to be infinitely differentiable [ 66 , 74 ] . The infinite differentiability of neural fields makes it possible to optimize loss functions that involve higher-order derivatives using gradient descent methods . Thus , unlike mesh-based geometry processing algorithms that approximate these objectives using surface discretizations , we can directly optimize in terms of the derivatives of the field . Our formulation provides a proof of concept that geometry processing can be done entirely using neural fields . In particular , we focus on two tasks : shape filtering ( e.g. , smoothing and sharpening ) and topology-preserving deformations . These two tasks are not only essential for downstream applications but also bring out the known challenges associated with implicit representations . We first tackle shape sharpening and smoothing and show how these can be done by optimizing a loss function based on geometric properties computed via the neural field ’ s derivatives . To achieve shape deformation , we propose to warp the neural field using a deformation vector field modeled by an invertible neural network [ 7 ] . With this invertible deformation field , we derive a training objective that models the implicitly represented surfaces as elastic shells . We compare our method with well-established mesh-based baselines [ 70 , 76 ] . Our method can match the quality of the mesh-based counterparts without discretizing the surface . We hope that our work can inspire future generations of geometry processing algorithms using neural fields . 2 Related Work . This paper builds on two bodies of work : polygon mesh geometry processing and neural field representations . We will focus on prior work on two geometry processing tasks : shape filtering and topology-preserving deformation . Please refer to Botsch et al . [ 11 ] for further reading . Mesh smoothing ( and sharpening ) . The goal of mesh smoothing is to remove high-frequency noise and produce a smooth surface . Noise removal can be achieved by Fourier analysis [ 76 , 79 ] or by modeling surface motion with a diffusion equation [ 26 ] . Smoothness can be promoted by minimizing energy functionals inspired by differential operators [ 50 , 72 , 82 ] . Sharpening is the inverse of smoothing . In this paper , we adopt the goal of smoothing from prior work [ 20 , 76 ] and design a corresponding differentiable objective for neural fields . Implicit field smoothing . Implicit representations can be smoothed by evolving the level set according to its curvature normal [ 49 , 54 ] . Many network architectures , such as ReLU MLP , bias toward smoothness when modeling an implicit field [ 61 , 66 , 74 ] . Regularization objectives and initialization schemes have been proposed to obtain smooth level sets from scanned surface points [ 5 , 31 ] . These works mostly use smoothness as a prior when training neural fields . Our work formulates an objective that allows smoothing or sharpening a neural field as desired . Mesh deformation . Mesh deformation is done by either deforming the shape ( i.e . vertices ) [ 9 , 69 ] or the space [ 6 , 36 , 41 , 48 , 65 ] . These algorithms usually draw inspiration from physics and differential geometry to develop energy functions that encourage natural deformation [ 10 , 15 , 24 , 70 , 71 , 77 ] . They also linearize such objectives so that they can be solved efficiently using linear solvers [ 9 ] . Recent research also applies deep learning to optimize for non-linear deformation losses [ 32 , 34 , 38 , 80 , 81 , 83 ] . Deforming a mesh , however , requires estimating the deformation objectives with spatial discretization and maintaining vertex connectivity in a way that preserves topology [ 42 ] . We deform neural fields with a continuous invertible field to circumvent these issues . Implicit field deformation . Prior works have studied the deformation of implicit fields for tasks like physics simulation [ 39 , 51 , 54 , 55 ] . These works use voxels or octrees to represent implicit fields , while our paper advocates for using neural fields for deformation . There is some past work on deforming neural fields to match a target shape [ 12 , 18 , 37 , 53 , 58 ] or image [ 43 , 57 , 60 ] without any correspondences between the two . In contrast , we perform such deformation with very few localized user-provided correspondences , but without any other information about the target shape . Our setting is very convenient for artists , but it requires the algorithm to provide a strong surface prior due to the lack of a densely specified target state . Other prior works on editing neural fields are trained on a dataset of shapes from a particular object category [ 25 , 29 , 30 , 33 , 85 ] . Since surface priors are specified implicitly by the dataset , it is unclear how to apply these methods to out-of-distribution shapes . In contrast , our paper requires no training dataset and instead enforces priors inspired by more general physical properties such as smoothness and elasticity . Remelli et al . [ 62 ] developed a way to differentiate iso-surface extraction , which allows deforming neural fields using objectives carried over from mesh-based algorithms . Our method avoids iso-surface extraction entirely and formulates differentiable deformation objectives directly on the implicit field . 3 Computing Surface Properties of Neural Fields . In this section , we will discuss how to compute surface properties using neural fields and establish notation for the following sections . Let ∂Ω be a surface enclosing the region Ω . A signed distance field ( SDF ) for surface ∂Ω is defined as d ( x ) = s ( x ) miny∈∂Ω ∥x− y∥ . The sign function s ( x ) evaluates to −1 if x ∈ Ω and 1 otherwise . We will assume that a neural field approximate an SDF whose zero-isosurface represents the surface of interest . For a neural field f , we denote its zero-isosurface using Mf . This section summarizes how to compute surface normals and curvatures from neural fields . This is well-understood and we summarize this here for convenience [ 39 , 54 , 59 ] . Surface normal . The surface normal is the vector perpendicular to the tangent plane and describes the local orientation of the surface . For an SDF , the outward oriented surface normal of its level set is the field gradient : nd ( x ) = ∇xd ( x ) . It can be shown that the SDF ’ s gradient norm is always one : ∥∇xd ( x ) ∥ = 1 . This property can be used to ensure that the field remains a valid SDF throughout any manipulation [ 4 , 31 ] . The closest point in ∂Ω from x can be found by x − d ( x ) nd ( x ) . This property can be used to sample points from the isosurface without creating a mesh [ 5 , 13 ] . Finally , since the tangent plane is perpendicular to the surface normal , we can project vectors onto the tangent plane by subtracting their projection onto the normal using the following projection matrix x : Pd ( x ) = I− nd ( x ) nd ( x ) T . This projection matrix allows us to characterize the tangent plane . This matrix is important for measuring how much a given deformation stretches the tangent direction [ 35 ] . Curvature . Intuitively , curvature describes how much a surface deviates from a plane . This can be captured by the total derivative of the surface normal , which is also known as the shape operator : SD ( x ) = Dnd ( x ) . For an SDF , the normal itself is given by the derivative of the field , thus the shape operator for an SDF is the Hessian of the field function : Hd ( x ) = D2d ( x ) . The shape operator can be used to capture many different notions of curvature , including mean curvature ( half the trace of SD ( x ) , denoted by κ̄ ) , Gaussian curvature ( the determinant of SD ( x ) ) , and principal curvatures ( the eigenvalues of SD ( x ) ) . We will use these to define objectives that smooth or sharpen the surface .

Summary: This work introduces a framework for geometry processing on surfaces represented as neural networks (neural implicit fields).  Namely, authors design two algorithms: one for surface smoothing, and one for surface deformation with control points (deformation handles). The core idea behind both is to introduce a new neural network (for smoothing another neural implicit field and for deformation an "invertible deformation field"), and optimize it with gradient descent, using a set of losses defined directly on implicit surface. Qualitative experimental evaluation is conducted on a set of meshes, for deformation comparison is provided to a single baseline: as-rigid-as-possible deformation, and for smoothing to Taubin smoothing.