Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields

1 INTRODUCTION . Neural implicit functions have emerged as a powerful tool for representing a variety of signals . Compared to conventional discrete representations , neural implicits are continuous and thus not tied to a specific resolution . Recently , neural implicits have gained significant attraction in a variety of applications ranging from 3D reconstruction ( Sitzmann et al. , 2019 ; Niemeyer et al. , 2020 ; Yariv et al. , 2020 ; Peng et al. , 2020 ) , neural rendering ( Mildenhall et al. , 2020 ; Pumarola et al. , 2020 ) , image translation ( Skorokhodov et al. , 2020 ; Chen et al. , 2020a ) to deformation approximation ( Deng et al. , 2019 ) . In this paper , we focus on neural implicit representations for 3D geometry . While neural implicits can theoretically model geometry with infinite resolution , in practice the output resolution is dependent on the representational power of neural nets . So far , the research community approaches the problem from two main directions . The first is to partition the implicit function using spatial structures ( Chabra et al. , 2020 ; Jiang et al. , 2020 ; Liu et al. , 2020 ; Takikawa et al. , 2021 ) , thus making the memory and computation demands dependent on the geometric complexity . The other direction focuses on improving networks ’ ability to represent high-frequency signals , either in a preprocessing step ( referred to as positional encoding ) ( Mildenhall et al. , 2020 ) or by using sinusoidal representation networks ( SIREN ) ( Sitzmann et al. , 2020 ) . However , training these networks is very challenging , as they are prone to overfitting and optimization local minima . Inspired by the classic computer graphics technique , displacement mapping ( Cook , 1984 ; Cook et al. , 1987 ) , we propose a novel parameterization of neural implicit functions , implicit displacement field , abbreviated as IDF , to circumvent the above issues . Our method automatically disentangles a given detailed shape into a coarse base shape represented as a continuous , low-frequency signed distance function and a continuous highfrequency implicit displacement field , which offsets the base iso-surface along the normal direction . The key novelty of our approach lies in extending the classic displacement mapping , which is discrete and lies only on the base surface , to a continuous function in the R3 domain and incorporating it into contemporary neural implicit representations , ergo achieving a disentanglement of geometric details in an unsupervised manner . Our main technical contribution includes 1. a principled and theoretically grounded extension of explicit discrete displacement mapping to the implicit formulation , 2. a neural architecture that creates a geometrically interpretable frequency hierarchy in the neural implicit shape representation by exploiting the inductive bias of SIRENs , and 3. introducing transferable implicit displacement fields by replacing the common coordinates input with carefully constructed transferrable features , thus opening up new opportunities for implicit geometry manipulation and shape modeling . Systematic evaluations show that our approach is significantly more powerful in representing geometric details , while being lightweight and highly stable in training . 2 RELATED WORK . Hierachical neural implicit shape representation . Neural implicit shape representation was initially proposed by several works concurrently ( Park et al. , 2019 ; Chen & Zhang , 2019 ; Mescheder et al. , 2019 ) , and since then many works have sought to introduce hierarchical structures into the neural representation for better expressiveness and generalizability . The majority of these methods focus on spatial structures . Chabra et al . ( 2020 ) ; Saito et al . ( 2019 ; 2020 ) use sparse regular voxels and dense 2D grid , respectively , to improve detail reconstruction . In the spirit of classic approaches , e.g . Frisken et al . ( 2000 ) ; Ohtake et al . ( 2003 ) , Liu et al . ( 2020 ) ; Takikawa et al . ( 2021 ) ; Martel et al . ( 2021 ) store learned latent codes in shape-adaptive octrees , leading to significantly higher reconstruction quality and increased rendering speed . A common disadvantage of these methods is that the memory use and model complexity are directly tied to the desired geometric resolution . In parallel , other proposed methods learn the spatial partition . Some of these methods decompose the given shape using parameterized templates , such as anisotropic Gaussians ( Genova et al. , 2019 ) , convex shape CVXNet ( Deng et al. , 2020 ; Chen et al. , 2020b ) or simple primitives ( Hao et al. , 2020 ) , while others represent local shapes with small neural networks and combine them together either using Gaussians ( Genova et al. , 2020 ) or surface patches ( Tretschk et al. , 2020 ) . Due to limitations of template functions and delicate spatial blending issues , these methods can only handle very coarse geometries . Concurrently , Li & Zhang ( 2021 ) propose a two-level neural signed distance function for singleview reconstruction . Exploiting the fact that most man-made shapes have flat surfaces , it represents a given shape as a coarse SDF plus a frontal and rear implicit displacement map for better detail construction . Besides having entirely different applications – we focus on representing significantly higher geometry resolutions – our implicit displacement is grounded in geometry principles and applies to general shapes . High-frequency representation in neural networks As formally explained by Xu ( 2018 ) ; Xu et al . ( 2019 ) ; Rahaman et al . ( 2019 ) ; Basri et al . ( 2020 ) , neural networks have a tendency to learn low-frequency functions . To combat this issue , Mildenhall et al . ( 2020 ) incorporate “ positional encoding ” for neural rendering and demonstrate remarkable progress in terms of detail reconstruction , which is a sinusoidal mapping for the input signal , a practice later theoretically justified by Tancik et al . ( 2020 ) . Alternatively , SIREN also shows impressive advances in detail representation by replacing ReLU activation with sin functions . With these new networks gaining popularity , a few works delve deeper and apply a coarse-to-fine frequency hierarchy in the training process for deformable shape representation ( Park et al. , 2020b ) and meshing ( Hertz et al. , 2021 ) . In our method , we also create a frequency hierarchy by leveraging this new form of networks – not only in the training scheme but also explicitly in the construction of the networks to reflect our geometry-motivated design principles . Detail transfer Detail transfer refers to transplanting the disentangled geometric details from a source shape onto a target object with high fidelity and plausibility . Classic detail transfer methods represent surface details as normal displacements ( Botsch et al. , 2010 ; Zhou et al. , 2007 ; Sorkine & Botsch , 2009 ) . The majority of them are parametric ( Ying et al. , 2001 ; Biermann et al. , 2002 ; Sorkine et al. , 2004 ; Zhou et al. , 2006 ; Takayama et al. , 2011 ) , relying on a consistent surface parameterization between the source and the target shape . Non-parametric approaches ( Chen et al. , 2012 ; Berkiten et al. , 2017 ) , on the other hand , find best-matching surface patches between the source and target , and copy the details iteratively from coarse to fine . These classic approaches produce high quality results , but often require a pre-defined base surface or abundant user inputs . In the “ deep ” realm , DeepCage ( Yifan et al. , 2020 ) proposed a neural deformation method that maps solely the coarse geometry , hence allowing detail transfer without tackling detail disentanglement . Hertz et al . ( 2020 ) learn the coarse-to-detail correspondence iteratively from multi-scale training data , while Chen et al . ( 2021 ) synthesizes details by upsampling a coarse voxel shape according to a style code of another shape using GANs . All of these approaches use explicit representations , hence they are subject to self-intersection and resolution limitations . D2IM-Net ( Li & Zhang , 2021 ) uses two planar displacement maps to transfer surface details by mapping the coordinates of the source and target shapes using part segementation , thus limiting the application to man-made rigid shapes . In comparison , our method does not require any correspondence mapping . 3 METHOD We represent a shape with fine geometric details using two SIREN networks of different frequencies in the activation functions . The SIREN with lower frequency describes a smooth base surface ; the SIREN with higher frequency adds microstructure to the base iso-surface by producing an implicit displacement field along the base ’ s normal direction ( see Figure 2 ) . In this section , we first formally define implicit displacement field by generalizing the classic explicit and discrete displacement mapping in Sec 3.1 , then in Sec 3.2 we introduce the network architectures and training strategies that are tailored to this definition , finaly in Sec 3.3 we extend the implicit displacement to address transferability . 3.1 IMPLICIT DISPLACEMENT FIELDS . In classic displacement mapping as shown in Figure 1 , high-frequency geometric details are obtained on a smooth base surface by taking samples from the base surface and offsetting them along their normal directions by a distance obtained ( with interpolation ) from a discrete height map . Two elements in this setting impede a direct adaptation for implicit shape representation : 1. the displacement mapping is defined discretely and only on the base surface , whereas implicit surface functions are typically defined continuously on the R3 domain ; 2. the base surface is known and fixed , whereas our goal is to learn the base surface and the displacement jointly on-the-fly . Addressing the above challenges , we first define implicit displacement fields ( IDF ) , which are continuous analog to height maps that extend displacement mapping to the R3 domain . Definition 1 . Given two signed distance functions f and f̂ and their respective iso-surfaces at a given value τ ∈ R , Sτ = { x ∈ R3|f ( x ) = τ } and Ŝτ = { x ∈ R3|f̂ ( x ) = τ } , an implicit displacement field d : R3 → R defines the deformation from Sτ to Ŝτ such that f ( x ) = f̂ ( x + d ( x ) n ) , where n = ∇f ( x ) ‖∇f ( x ) ‖ . ( 1 ) This definition is schematically illustrated in Figure 3 , where the iso-surface S0 and Sτ are mapped to Ŝ0 and Ŝτ with the same implicit displacement field d. Notably , the height map in classic displacement mapping is a discrete sampling of IDF for the limited case τ = 0 . In the context of surface decomposition , our goal is to estimate the base surface f and the displacement d given an explicitly represented detailed surface Ŝ0 . Following equation 1 , we can do so by minimizing the difference between the base and the ground truth signed distance at query points x ∈ R3 and their displaced position x̂ = x + d ( x ) n , i.e. , min |f ( x ) − f̂GT ( x̂ ) | . However , this solution requires evaluating f̂GT ( x̂ ) dynamically at variable positions x̂ , which is a costly operation as the detailed shapes are typically given in explicit form , e.g. , as point clouds or meshes . Hence , we consider the inverse implicit displacement field d̂ , which defines a mapping from Ŝτ to Sτ , f ( x̂ + d̂ ( x̂ ) n ) = f̂ ( x̂ ) , as depicted in Figure 3 . Assuming the displacement distance is small , we can approximate n , the normal after inverse displacement , with that before the inverse displacement , i.e . f ( x̂ + d̂ ( x̂ ) n̂ ) = f̂ ( x̂ ) , where n̂ = ∇f ( x̂ ) ‖∇f ( x̂ ) ‖ . ( 2 ) This is justified by the following theorem and corollary , which we prove in the Appendix A. Theorem 1 . If function f : Rn → R is differentiable , Lipschitz-continuous with constant L and Lipschitz-smooth with constant M , then ‖∇f ( x + δ∇f ( x ) ) −∇f ( x ) ‖ ≤ |δ|LM . Corollary 1 . If a signed distance function f satisfying the eikonal equation up to error > 0 , |‖∇f‖ − 1| < , is Lipschitz-smooth with constant M , then ‖∇f ( x + δ∇f ( x ) ) − ∇f ( x ) ‖ < ( 1 + ) |δ|M . Given n = ∇f ( x ) ‖∇f ( x ) ‖ , n̂ = ∇f ( x̂ ) ‖∇f ( x̂ ) ‖ , and x̂ = x + d ( x ) n , let δ = d ( x ) ‖∇f ( x ) ‖ , we can show ‖n̂− n‖ ≤ 1 + 1− |δ|M ( c.f . Appendix A ) . In other words , the difference of n̂ and n is bounded by a small constant . Thus we obtain the approximation in equation 2 , which allows us to presample training samples { x̂ } and use precomputed f̂GT ( x̂ ) or its derivatives ( see Sec 3.2 ) for supervision .

Summary: This paper proposes an implicit neural representation for 3D geometry, named implicit displacement fields. The method is consists of two SIREN networks as baseline models to approximate: a coarse base shape representation (low-frequency) and an implicit displacement field (for high-frequency details). Different from previous works, the displacement field is implicitly represented by a SIREN network. The coarse and fine networks design also enables applications such as shape manipulation.