Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields

1 INTRODUCTION . Neural implicit functions have emerged as a powerful tool for representing a variety of signals . Compared to conventional discrete representations , neural implicits are continuous and thus not tied to a specific resolution . Recently , neural implicits have gained significant attraction in a variety of applications ranging from 3D reconstruction ( Sitzmann et al. , 2019 ; Niemeyer et al. , 2020 ; Yariv et al. , 2020 ; Peng et al. , 2020 ) , neural rendering ( Mildenhall et al. , 2020 ; Pumarola et al. , 2020 ) , image translation ( Skorokhodov et al. , 2020 ; Chen et al. , 2020a ) to deformation approximation ( Deng et al. , 2019 ) . In this paper , we focus on neural implicit representations for 3D geometry . While neural implicits can theoretically model geometry with infinite resolution , in practice the output resolution is dependent on the representational power of neural nets . So far , the research community approaches the problem from two main directions . The first is to partition the implicit function using spatial structures ( Chabra et al. , 2020 ; Jiang et al. , 2020 ; Liu et al. , 2020 ; Takikawa et al. , 2021 ) , thus making the memory and computation demands dependent on the geometric complexity . The other direction focuses on improving networks ’ ability to represent high-frequency signals , either in a preprocessing step ( referred to as positional encoding ) ( Mildenhall et al. , 2020 ) or by using sinusoidal representation networks ( SIREN ) ( Sitzmann et al. , 2020 ) . However , training these networks is very challenging , as they are prone to overfitting and optimization local minima . Inspired by the classic computer graphics technique , displacement mapping ( Cook , 1984 ; Cook et al. , 1987 ) , we propose a novel parameterization of neural implicit functions , implicit displacement field , abbreviated as IDF , to circumvent the above issues . Our method automatically disentangles a given detailed shape into a coarse base shape represented as a continuous , low-frequency signed distance function and a continuous highfrequency implicit displacement field , which offsets the base iso-surface along the normal direction . The key novelty of our approach lies in extending the classic displacement mapping , which is discrete and lies only on the base surface , to a continuous function in the R3 domain and incorporating it into contemporary neural implicit representations , ergo achieving a disentanglement of geometric details in an unsupervised manner . Our main technical contribution includes 1. a principled and theoretically grounded extension of explicit discrete displacement mapping to the implicit formulation , 2. a neural architecture that creates a geometrically interpretable frequency hierarchy in the neural implicit shape representation by exploiting the inductive bias of SIRENs , and 3. introducing transferable implicit displacement fields by replacing the common coordinates input with carefully constructed transferrable features , thus opening up new opportunities for implicit geometry manipulation and shape modeling . Systematic evaluations show that our approach is significantly more powerful in representing geometric details , while being lightweight and highly stable in training . 2 RELATED WORK . Hierachical neural implicit shape representation . Neural implicit shape representation was initially proposed by several works concurrently ( Park et al. , 2019 ; Chen & Zhang , 2019 ; Mescheder et al. , 2019 ) , and since then many works have sought to introduce hierarchical structures into the neural representation for better expressiveness and generalizability . The majority of these methods focus on spatial structures . Chabra et al . ( 2020 ) ; Saito et al . ( 2019 ; 2020 ) use sparse regular voxels and dense 2D grid , respectively , to improve detail reconstruction . In the spirit of classic approaches , e.g . Frisken et al . ( 2000 ) ; Ohtake et al . ( 2003 ) , Liu et al . ( 2020 ) ; Takikawa et al . ( 2021 ) ; Martel et al . ( 2021 ) store learned latent codes in shape-adaptive octrees , leading to significantly higher reconstruction quality and increased rendering speed . A common disadvantage of these methods is that the memory use and model complexity are directly tied to the desired geometric resolution . In parallel , other proposed methods learn the spatial partition . Some of these methods decompose the given shape using parameterized templates , such as anisotropic Gaussians ( Genova et al. , 2019 ) , convex shape CVXNet ( Deng et al. , 2020 ; Chen et al. , 2020b ) or simple primitives ( Hao et al. , 2020 ) , while others represent local shapes with small neural networks and combine them together either using Gaussians ( Genova et al. , 2020 ) or surface patches ( Tretschk et al. , 2020 ) . Due to limitations of template functions and delicate spatial blending issues , these methods can only handle very coarse geometries . Concurrently , Li & Zhang ( 2021 ) propose a two-level neural signed distance function for singleview reconstruction . Exploiting the fact that most man-made shapes have flat surfaces , it represents a given shape as a coarse SDF plus a frontal and rear implicit displacement map for better detail construction . Besides having entirely different applications – we focus on representing significantly higher geometry resolutions – our implicit displacement is grounded in geometry principles and applies to general shapes . High-frequency representation in neural networks As formally explained by Xu ( 2018 ) ; Xu et al . ( 2019 ) ; Rahaman et al . ( 2019 ) ; Basri et al . ( 2020 ) , neural networks have a tendency to learn low-frequency functions . To combat this issue , Mildenhall et al . ( 2020 ) incorporate “ positional encoding ” for neural rendering and demonstrate remarkable progress in terms of detail reconstruction , which is a sinusoidal mapping for the input signal , a practice later theoretically justified by Tancik et al . ( 2020 ) . Alternatively , SIREN also shows impressive advances in detail representation by replacing ReLU activation with sin functions . With these new networks gaining popularity , a few works delve deeper and apply a coarse-to-fine frequency hierarchy in the training process for deformable shape representation ( Park et al. , 2020b ) and meshing ( Hertz et al. , 2021 ) . In our method , we also create a frequency hierarchy by leveraging this new form of networks – not only in the training scheme but also explicitly in the construction of the networks to reflect our geometry-motivated design principles . Detail transfer Detail transfer refers to transplanting the disentangled geometric details from a source shape onto a target object with high fidelity and plausibility . Classic detail transfer methods represent surface details as normal displacements ( Botsch et al. , 2010 ; Zhou et al. , 2007 ; Sorkine & Botsch , 2009 ) . The majority of them are parametric ( Ying et al. , 2001 ; Biermann et al. , 2002 ; Sorkine et al. , 2004 ; Zhou et al. , 2006 ; Takayama et al. , 2011 ) , relying on a consistent surface parameterization between the source and the target shape . Non-parametric approaches ( Chen et al. , 2012 ; Berkiten et al. , 2017 ) , on the other hand , find best-matching surface patches between the source and target , and copy the details iteratively from coarse to fine . These classic approaches produce high quality results , but often require a pre-defined base surface or abundant user inputs . In the “ deep ” realm , DeepCage ( Yifan et al. , 2020 ) proposed a neural deformation method that maps solely the coarse geometry , hence allowing detail transfer without tackling detail disentanglement . Hertz et al . ( 2020 ) learn the coarse-to-detail correspondence iteratively from multi-scale training data , while Chen et al . ( 2021 ) synthesizes details by upsampling a coarse voxel shape according to a style code of another shape using GANs . All of these approaches use explicit representations , hence they are subject to self-intersection and resolution limitations . D2IM-Net ( Li & Zhang , 2021 ) uses two planar displacement maps to transfer surface details by mapping the coordinates of the source and target shapes using part segementation , thus limiting the application to man-made rigid shapes . In comparison , our method does not require any correspondence mapping . 3 METHOD We represent a shape with fine geometric details using two SIREN networks of different frequencies in the activation functions . The SIREN with lower frequency describes a smooth base surface ; the SIREN with higher frequency adds microstructure to the base iso-surface by producing an implicit displacement field along the base ’ s normal direction ( see Figure 2 ) . In this section , we first formally define implicit displacement field by generalizing the classic explicit and discrete displacement mapping in Sec 3.1 , then in Sec 3.2 we introduce the network architectures and training strategies that are tailored to this definition , finaly in Sec 3.3 we extend the implicit displacement to address transferability . 3.1 IMPLICIT DISPLACEMENT FIELDS . In classic displacement mapping as shown in Figure 1 , high-frequency geometric details are obtained on a smooth base surface by taking samples from the base surface and offsetting them along their normal directions by a distance obtained ( with interpolation ) from a discrete height map . Two elements in this setting impede a direct adaptation for implicit shape representation : 1. the displacement mapping is defined discretely and only on the base surface , whereas implicit surface functions are typically defined continuously on the R3 domain ; 2. the base surface is known and fixed , whereas our goal is to learn the base surface and the displacement jointly on-the-fly . Addressing the above challenges , we first define implicit displacement fields ( IDF ) , which are continuous analog to height maps that extend displacement mapping to the R3 domain . Definition 1 . Given two signed distance functions f and f̂ and their respective iso-surfaces at a given value τ ∈ R , Sτ = { x ∈ R3|f ( x ) = τ } and Ŝτ = { x ∈ R3|f̂ ( x ) = τ } , an implicit displacement field d : R3 → R defines the deformation from Sτ to Ŝτ such that f ( x ) = f̂ ( x + d ( x ) n ) , where n = ∇f ( x ) ‖∇f ( x ) ‖ . ( 1 ) This definition is schematically illustrated in Figure 3 , where the iso-surface S0 and Sτ are mapped to Ŝ0 and Ŝτ with the same implicit displacement field d. Notably , the height map in classic displacement mapping is a discrete sampling of IDF for the limited case τ = 0 . In the context of surface decomposition , our goal is to estimate the base surface f and the displacement d given an explicitly represented detailed surface Ŝ0 . Following equation 1 , we can do so by minimizing the difference between the base and the ground truth signed distance at query points x ∈ R3 and their displaced position x̂ = x + d ( x ) n , i.e. , min |f ( x ) − f̂GT ( x̂ ) | . However , this solution requires evaluating f̂GT ( x̂ ) dynamically at variable positions x̂ , which is a costly operation as the detailed shapes are typically given in explicit form , e.g. , as point clouds or meshes . Hence , we consider the inverse implicit displacement field d̂ , which defines a mapping from Ŝτ to Sτ , f ( x̂ + d̂ ( x̂ ) n ) = f̂ ( x̂ ) , as depicted in Figure 3 . Assuming the displacement distance is small , we can approximate n , the normal after inverse displacement , with that before the inverse displacement , i.e . f ( x̂ + d̂ ( x̂ ) n̂ ) = f̂ ( x̂ ) , where n̂ = ∇f ( x̂ ) ‖∇f ( x̂ ) ‖ . ( 2 ) This is justified by the following theorem and corollary , which we prove in the Appendix A. Theorem 1 . If function f : Rn → R is differentiable , Lipschitz-continuous with constant L and Lipschitz-smooth with constant M , then ‖∇f ( x + δ∇f ( x ) ) −∇f ( x ) ‖ ≤ |δ|LM . Corollary 1 . If a signed distance function f satisfying the eikonal equation up to error > 0 , |‖∇f‖ − 1| < , is Lipschitz-smooth with constant M , then ‖∇f ( x + δ∇f ( x ) ) − ∇f ( x ) ‖ < ( 1 + ) |δ|M . Given n = ∇f ( x ) ‖∇f ( x ) ‖ , n̂ = ∇f ( x̂ ) ‖∇f ( x̂ ) ‖ , and x̂ = x + d ( x ) n , let δ = d ( x ) ‖∇f ( x ) ‖ , we can show ‖n̂− n‖ ≤ 1 + 1− |δ|M ( c.f . Appendix A ) . In other words , the difference of n̂ and n is bounded by a small constant . Thus we obtain the approximation in equation 2 , which allows us to presample training samples { x̂ } and use precomputed f̂GT ( x̂ ) or its derivatives ( see Sec 3.2 ) for supervision .

Summary: The paper proposes a an implicit representation of surface displacement fields that is well learnable by neural networks. The network is based on two SIREN architectures that separate low and high frequency information in the model. One returns an SDF of the rough shape while the second returns a displacement field at the surface of the SDF. In addition to this architecure (which existed before but for a different application), the paper introduces an inverse implicit mapping that speeds up the evaluation of the loss, and a feature based UV map that can be transferred between similar shapes. The paper compares to previous work on SketchFab, and specifically evaluates against a vanilla SIREN model where this method results in both a smaller model size as well as more detailed reconstruction. Additionally, there are experiments that show how details can be transferred across different (but similar) models.