Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields
Authors: Wang Yifan
Lukas Rahmann
Olga Sorkine-hornung
Venue: ICLR 2022 Poster (2022)
Keywords: implicit functions
shape reconstruction
shape representation
object reconstruction

Abstract: We present implicit displacement fields, a novel representation for detailed 3D geometry. Inspired by a classic surface deformation technique, displacement mapping, our method represents a complex surface as a smooth base surface plus a displacement along the base's normal directions, resulting in a frequency-based shape decomposition, where the high-frequency signal is constrained geometrically by the low-frequency signal. Importantly, this disentanglement is unsupervised thanks to a tailored architectural design that has an innate frequency hierarchy by construction. We explore implicit displacement field surface reconstruction and detail transfer
and demonstrate superior representational power, training stability, and generalizability.

GEOMETRY-CONSISTENT NEURAL SHAPE
REPRESENTATION WITH IMPLICIT DISPLACEMENT FIELDS
Wang Yifan
Lukas Rahmann
Olga Sorkine-Hornung ETH Zurich acrkine@inf .et.z. ch
ETH Zurich
ETH Zurich lukas. att. com com
ywangêinf.ed www.:eiiffehhh....mmmmmm wt....thhcc.o............ ch
ABSTRACT
We present implicit displacement fields, novel representation for detailed 3D ge- ometry. Inspired by classic surface deformation technique, displacement map- ping, method represents complex surface smooth base surface plus a displacement along the base's normal directions, resulting in a frequency-basee shape decomposition, where the high-frequency signal is constrained geometri- cally by the low-frequency signal. Importantly, this disentanglement is unsu- pervised thanks hierarchy by construction. We explore implicit displacement field surface re- construction and detail transfer and demonstrate superior representational power, training stability, and generalizability. Code and data available at: https: //github. http://ttooo/iiiiie/e
tailored architectural design that has an innate frequency
INTRODUCTION
Neural implicit functions have emerged as a powerful tool for representing a variety of signals. Compared to conventional discrete representations, neural implicits are continuous and thus not tied
to specific resolution. Recently, neural implicits have gained significant attraction in variety of
applications ranging from 3D reconstruction (Sitzmann et al. [2019] Niemeyer et al.] [0200] Yariv
et al., 2020 Peng et al. (200)). neural rendering (Mildenhall et al. 2020; Pumarola et al. (020). im-
age translation (Skorokhodov al. 2020, Chen et al. (020a) to deformation approximation (Deng
et al. 2019). In this paper, we focus on neural implicit representations for 3D geometry.
While neural implicits can theoretically model geometry with infinite resolution, in prac- tice the output resolution is dependent the representational power of neural nets. So far, research community approaches the problem from two main directions. The first is to partition the implicit function using spatial structures (Chabra et al., 2020;; Diang al. 2020 Liu et al.] 2020; Takikaaa et al. 2021). thus making the memory and computation demands dependent the geometric complexity. The other direction focuses on improving networks' ability to represent freuuniry signals, either in preprocess- ing step (referred to as positional encoding) (Mildenhall al. 2020) by using sinusoidal representation net- works (SIREN) (Sitzmann et al. (2220). However, training these networks is very challenging, as they are prone overfitting and optimization local minima.
Inspired by the classic computer graphics technique, dis- placement mapping (Cook) 1984, Cook et al. 1987). ) we propose a novel parameterization tions, implicit displacement field, abbreviated as IDF, to circumvent the above issues. Our method automatically disentangles given detailed shape into coarse base shape represented as a continuous, low- cow-frequency signed distance function and a continuous high- frequency implicit displacement field, which offsets the base iso-surface along the normal direction.
Figure I: Displacement mapping in ID. The dctailed surface (upper blue) is ereated by offsetting samples of base surface (upper black) using the height map shown below.
neural implicit func-

---

The key novelty of our approach lies in extending the classic displacement mapping, which is dis- crete and lies only on the base surface, to a continuous function in the R domain and incorporating it into contemporary neural implicit representations, ergo achieving disentanglement of geometric details in an unsupervised manner.
Our main technical contribution includes
1. a principled and theoretically grounded extension of explicit discrete displacement mapping to the implicit formulation,
2. a neural architecture that creates a geometrically interpretable frequency hierarchy in the neural implicit shape representation by exploiting the inductive bias of SIRENS, and
3. introducing transferable implicit displacement fields by replacing the common coordinates imput with carefully constructed transferrable features, thus opening up new opportunities for implicit geometry manipulation and shape modeling.
Systematic evaluations show that our approach significantly more powerful representing geo- metric details, while being lightweight and highly stable in training.
2 RELATED WORK
Hierachical neural implicit shape representation. Neural implicit shape representation was initially proposed by several works concurrently (Park al. 2019 Chen & Zhang 2019 Mescheder et al.] (209)), and since then many works have sought to introduce hierarchical structures into the neural representation for better expressiveness and generalizability. The majority of these methods focus on spatial structures. Chabra et al. (220)); Saito et al. 2019 2020) use sparse regular voxels and dense 2D grid, respectively, to improve detail reconstruction. In the spirit of classic approaches, e.g. Frisken et al. (2000); Ohtake et al. (200)). Liu et al.) (2020): Takiiaaa et al. (2021) ) Martel cct al. (2021) store learned latent codes shape-adaptive octrees, leading to significantly higher reconstruction quality and increased rendering speed. A common disadvantage of these methods that the memory use and model complexity are directly tied to the desired geometric resolution. In parallel, other proposed methods learn the spatial partition. Some of these methods decompose the given shape using parameterized templates, such as anisotropic Gaussians (Genova et al.) 2019)) convex shape CVXNet (Deng al. 2020) Chen al. (020b) simple primitives (Hao et al.) 2020), while others represent local shapes with small neural networks and combine them together either using Gaussians (Genova et al. (2020) surface patches (Tretschk et al. 2020).. Due to limitations of template functions and delicate spatial blending issues, these methods can only handle very coarse geometries.
Concurrently, Li & Zhang (2021) propose a two-level neural signed distance function for single- view reconstruction. Exploiting the fact that most man-made shapes have flat surfaces, represents given shape as a coarse SDF plus a frontal and rear implicit displacement map for better detail construction. Besides having entirely different applications we focus on representing significantly higher geometry resolutions our implicit displacement is grounded in geometry principles and applies to general shapes.
High-frequency representation in neural networks As formally explained by Xu 20188 Xu et al. (2019); Rahaman et al. ) Basri et al.] (2020), neural networks have a tendency to learn low-frequency functions. To combat this issue, Mildenhall al. (2020) incorporate "positional en- coding" for neural rendering and demonstrate remarkable progress in terms of detail reconstruction, which sinusoidal mapping for the input signal, a practice later theoretically justified by [Tan- cik et al. (2020). Alternatively, SIREN also shows impressive advances in detail representation by replacing ReLU activation with sin functions. With these new networks gaining popularity, a few works delve deeper and apply coarse-to-fine frequency hierarchy in the training process for de- formable shape representation (Park et al.) (020b) and meshing (Hertz et al.) 2021). )). In our method, also create frequency ing scheme but also explicitly in the construction of the networks to reflect our geometry geometry-motivate motivated design principles.
Trequency/hii hierarchy by leveraging this new form of networks not only in the train-
2

---

+
base SDF
impheit displacement
detailed SDF
Figure 2: Method overview. We represent detailed geometries as sum of base shape represented as low-frequency signed distance function a high-frequency implicit displacement field, which offsets the base 1so-surface along the base's normal directions.
Detail transfer Detail transfer refers transplanting the disentangled geometric details from
source shape onto target object with high fidelity and plausibility. Classic detail transfer methods
represent surface details as normal displacements (Botsch et al. [0100] Zhou et al. 2007 Sorkine
& Botsch 2009). The majority of them are parametric (Ying al. 2001] Biermann et al. 2002
Sorkine et al. 2004| Zhou et al. 2006 Takayama et al. 2011). relying on a consistent surface parameterization between the source and the target shape. Non- con-parametric approaches (Chen et al., 2012 Berkiten et al. 2017). on the other hand, find best-matching surface patches between the source and target, and copy the details iteratively from coarse to fine. These classic approaches produce high quality results, but often require a pre-defined base surface or abundant user inputs. In the "deep" realm, DeepCage (Yifan et al.] 2020) proposed neural deformation method that maps solely the coarse geometry, hence allowing detail transfer without tackling detail disentanglement. Hertz et al. (2020) learn the coarse-to-detall correspondence iteratively from multi-scale training data, while Chen et al. (2021) synthesizes details by upsampling a coarse voxel shape according to style code of another shape using GANs. All of these approaches use explicit representations, hence they are subject to eef-intersectton and resolution limitations. D²IM-Net (Li & Zhang 2021] uses two planar displacement maps to transfer surface details by mapping the coordinates of the source and target shapes using part segementation, thus limiting the application to man-made rigid shapes. In comparison, our method does not require any correspondence mapping.
3 METHOD
We represent shape with fine geometric details using two SIREN networks of different frequencies in the activa- tion functions. The SIREN with lower frequency deseribes smooth base surface; the SIREN with higher frequency adds microstructure to the base iso-surface by producing an implicit displacement field along the base's normal di- rection (see Figure 2).
implici dippaasmen da implist invese dislacement da
In this section, we first formally define implicit displace- ment field by generalizing the classic explicit and discrete displacement mapping in Sec 3.I then in Sec 3.2 we in- troduce the network architectures and training strategies that tailored to this definition, finaly in Sec [.3 we extend the implicit displacement to address transferability.
Figure 3: An implicit displacement field for ID-curve. The displacement is defined only on the zero-isosurface but also on arbitrary isosurfaces S,
3.1 IMPLICIT DISPLACEMENT FIELDS
In classic displacement mapping as shown in Figure [ high-frequency geometric details are ob- tained on a smooth base surface by taking samples from the base surface and offsetting them along their normal directions by distance obtained (with interpolation) from a discrete height map. Two elements in this setting impede a direct adaptation for implicit shape representation: I. the displace- ment mapping is defined discretely and only the base surface, whereas implicit surface functions are typically defined continuously on the 3 domain: the base surface known and fixed, whereas goal is to learn the base surface and the displacement, jointly on-the-fly.
3

---

Addressing the above challenges, first define implicit displacement fields (IDF), which are tinuous analog to height maps that extend displacement mapping to the R domain.
Definition Given two signed distance functions f and f and their respective iso-surfaces at given value E IR St = {x E [R = = and S, = {x (=*Iff()==>> T.. implicit
displacement field d: I2 + E. defines the deformation from S to Sr such that
Vf (x)
f (x) = f (x + d n)) where n
n),
(I)
This definition is schematically illustrated in Figure where the iso-surface So and S, are mapped to So and S, with the same implicit displacement field d. Notably, the height map in classic dis- placement mapping is discrete sampling of IDF for the limited case r 0.
In the context of surface decomposition, our goal is to estimate the base surface f and the displace- ment d given an explicitly represented detailed surface Sg Following equation we can do so by minimizing the difference between the base and the ground truth signed distance at query points R and their displaced position x + (x) n, i.e., min If (x) for (x) |.-
However, this solution requires evaluating for (x) dynamically at variable positions which is a costly operation the detailed shapes are typically given in explicit form, e.g., point clouds or meshes. Hence, consider the inverse implicit displacement field d, which defines mapping from S, to S.. f ( (x (x) = (x), as depicted in Figure
Assuming the displacement distance is small, we can approximate n, the normal after inverse dis- placement, with that before the inverse displacement, i.e.
f f(x + dx)) (&), where
V (x)
VV (x) I|
This is justified by the following theorem and corollary, which we prove the Appendix A
Theorem 1. If function f IR" Lipschitz-smonee smooth with constant M, then |/ (x 8V (x)) - V (x) S 18
is differentiable. Lipschirz-contimuuur with constant and
Corollary 1. If signed distance function J satisfying the eikonal equation up to error > 0. ||| 1| 6, Lipschitz-sman smooth with constant M, then [[V (x 8 (x)) V <
(1 M)
Given n = [V ñ = |V
f((x)
f(() and x = x ( let 8 = V we can show
|| nl s
1
18] M. (c.f. Appendix A). In other words, the difference of ñ and is bounded by small 1-e
constant. Thus we obtain the approximation in equation 2 which allows us to presample training samples (*) and use precomputed fer (x) its derivatives (see Sec 3.2 for supervision.
3.2 NETWORK DESIGN AND TRAINING
The formulation of (inverse) implicit field in the previous section is based on three assumptions: (i f is smooth, (ii) d is small, (iii) f satisfies the eikonal constraint up to error bound. In this section, we describe our network architecture and training technique, with emphasis on meeting these requirements.
Network architecture. We propose to model f and with two SIRENS denoted as N and Aoo, where wp and Wp refer to the frequency hyperparameter the sine activation functions x + sin (wx). Evidently, shown in Figure w dictates an upper bound on the frequencies the network capable of representing. thereby also determines the network's inductive bias for smoothness. Correspondingly, we enforce the smoothness of f and detail--epresenting capacity of d using a smaller wi and larger wp, e.g. Wp 15 and wp 60. Moreover, we add scaled tanh activation to the last linear layer of AdD, f.e. a tanh (-), which ensures that the displacement distance is smaller than the constant More insight about the choice of W is detailed in Section B.2
4

---

0.1
0.8
0.6
0.05
X
0.4
0.2
0
0
0.05
0.1
0.15
N (x)
Figure 4: Smoothness control via SIREN's frequency hyperpa- rameter Overfitting SIREN the right image with = 30 Figure S: Attenuation as function of base SDE
(first) and w (middle) shows that smaller Icads to a smoother result.
Networks containing high-freen freuerryyency signals, e.g. Aoo require large amounts of accurate ground truth data for supervision to avoid running into optimization local minima (Park et al. 2020b). Consequently, when dense and accurate ground truth SDF values are not available, high-frequency signals often create artifacts. This often the case in void regions when learning from point clouds, only implicit regularization and fuzzy supervision is applied (see the first and last terms of equa- tion 4). Hence, we apply an attenuation function X (A=")
to subdue A far
(0/ from the base surface, where v determines the speed of attenuation as depicted in Figure 5
Combining the aforementioned components, we can compute the signed distance of the detailed shape at query point x in two steps:
Vf Vx(x (x) IV (x)
f (x) N (x), (xx)=Nwr x + x AoD (x)
(3)
Training. We adopt the loss from SIREN, which constructed to learn SDFs directly from oriented point clouds by solving the eikonal equation with boundary constraint the on-surface points. Denoting the input domain Ω (by default as P = {(p,, n,)). the loss computed as equation where ((0.1.2.33 denote loss weights:
[-1, 1J and the ground truth point cloud
C = Exen 2o (Vf
+
Σ (p.n)EP
(Arlf + (I-(Vf(p).
Σ XERXP
exp (-100 f(x))
+
As the displacement and the attenuation functions depend on the base network, it is beneficial to have well-behaving base network when training the displacement (see Sec 4.2). Therefore, we adopt a progressive learning scheme, which first trains A/w,, and then gradually increase the impact of Awd Notably, similar frequency-based coarse-to-fine training techniques are shown to improve the optimization result recent works (Park et al. 2020b Hertz et al. 2021).
We implement the progressive training via symmetrically diminishinng/icceasinn learning rates and loss weights for the base/displacemmnt networks. For brevity. we describe the procedure for loss weights only, and apply the same to the learning rates in our implementation. First, train N by substituting ff in the loss equation 4 with resulting base-only loss denoted L,. Then, starting from a training percentile Tm. E [0, 1J. we combine L and & via f + (1 x) Cj with K
[ [+ cos n (===)))) )
where é E [I'm 1] denotes the current training progress.
2
3.3 TRANSFERABLE IMPLICIT DISPLACRMENT FIELD.
In classic displacement mapping, the displacement is queried by the UV-coordinates from surface parameterization, which makes the displacement independent of deformations of the base surface. We can achieve similar effect without parameterization by learning query features, which emulate the UV-coordinates deseribe the location of the 3D query points w.r.f. the base surface.
We construct the query features using two pieces of information: (i) a global context descriptor, (x), describing the location of the query point in relation to the base surface in semantically
5

---

point normals
Feature Extractor
(x))
M
(%.3)
Spanee Point Cloud
NoD
(x)
f(x
i
N
banh (2-)
TuD
Non-Imnseenhle IDF X
Transfembll IDF T
Figure 6: Illustrations for transferable and transferafeebe implicit fields. The transferable modules are pink and the shape-specific modules are yellow. Instead of consuming the euclidean coordinates, the transferable displacement network takes a maal-anddtrranslatin invariant feature as inputs, which describes the relative position of the query point to the base shape.
meaningful way. (ii) the base signed distance value f (x), which gives more precise relative location with respect to the base surface. Since both differentiable w.r.f the euclidean coordinates of the query point, we can still train N using derivatives as in equation
Our global context descriptor is inspired by Convolutional Occupancy Networks (Peng al. 2020) Specifically, we project the sparse on-surface point features obtained using a conventional point cloud encoder onto regular 3D (or 2D, c.f. Sec grid, then use a convolutional module prop- agate sparse on-surface point features to the off-surface area, finally obtain the query feature using bilinear interpolation. We use normals instead of point positions as the input to the point cloud encoder, making the features scale-invariant and translation- innatioonivvaiiant. Note that ideally the features should also be votation-invarto in fact generalize under small local rotational deformations, which is sufficient for transferring dis- placements between two roughly aligned shapes. We leave further explorations in this direction for future work.
Nevertheless, as we empirically show later, normal features can
Noo tasked to predict the displacement conditioning on (x) and (x). However, empiri- cal studies (Chan al. 2020) suggest that SIREN does not handle high-dimensional inputs well. Hence, we adopt the FiLM conditioning (Perez et al.] 2018; Dumoulin et al.) (2088) as suggested affine transformation to the features of each layer. Specifically, a mapping network M converts & (x) to a set of C- dimensional frequency modulators and phase shifters (Y, 3.), which transform the i-th linear layer to (1 7%) (W; x + b,) Brs where W, and b, the parameters in the linear layer and denotes element-wise multiplication. Finally, since SIREN assumes inputs in range (-1, scale f using f (x) = tanh ( (x)) to capture the variation close to the surface area, where is the attenuation parameter described section
by CChn et al. (2020), which feeds the conditioning latent vector
Figure 6 summarizes the difference between transferable and non-transferable displacement fields. Formally, the signed distance function of the detailed shape in equation can be rewritten as
=x))= = - (x+ xx+x(((x)) TWO (f (x), (
Vs fx))
(5)
| x
RESULTS
We now present the results of our method. In 4.c evaluate our networks in terms of geometric detail representation by comparing with tf-te-of-the-ar methods on the single shape fitting task. We then evaluate various design components in ablation study in Sec 4.2 Finally, validate the transferability the displacement fields in detail transfer task Sec 4.3 Extended qualitative and quantitative evaluations included in section
Implementation details. By default, both the base and the displacement nets have hidden lay- with 256 channels each. The maximal displacement a. attenuation factors v, and the switching training percentile is set to T,, are set to 0.05, 0.02 and 0.2 respectively; The loss weights (,0.1.23) in equation are set to 5, 400, 40 and 50. We train models for 120 epochs using ADAM opti-

---

Chemier distance poinis to distancee 10- nooraal cosine distance -10-2
Prognessive FFN
(LOD4)
(1.0D6)
E
60
w 30
SIREN-7
60
Residual
D-SDF
Ours
SketchFab-16
5.47/3.77
2.27/4.24
1
9.85/6.64
4.85/2.56
181/59.0
2.85/4.39
1.22//.25
Table I: Quantitative comparison. Among the benchmarking methods, only NGLOD LOD-6, using 256x number of parameters compared model, yield results close SIREN models with larger have convergence issues: despite our best efforts, the models still diverged most cases. Please refer to Table for a more comprehensive evaluation.
ground truth
ours (4.8MB)
NGL.OD-LODD6 (946MB)
NGLOD-LOOD (16MB)
D-SDF (7.4MB)
SIREN-7 30
(5.6MB)
Figure 7: Comparison of detail reconstruction (better viewed with zoom-in). We show the best 5 methods and their model sizes according to Table 1 more results provided in section B.1
mizer with initial learning rate of 0.0001 and decay to 0,00001 using cosine annealing (Loshchilov & Hutter, 2016) after finishing 80% of the training epochs. We presample million surface points per mesh for supervision. Each training iteration uses 4096 subsampled surface points and 4096 off- surface points uniformly sampled from the [-1, 1, 1]a bounding box. To improve the convergence rate, initialize SIREN models by pre-training the base model Aus (and baseline SIREN, c.f. Sec 4.1) to sphere with radius 0.5. This initialization is optional for our training but critical for baseline SIRENS.
Data. We test our method using 16 high-resolution shapes, including 14 from Sketchfab (ske) 2021) and 2 from Stanford 3DScanRepo (sta 2021). Our transferable displacement model is tested
using shapes provided by Berkiten et (011)) Yang al. (2020). and Zhou & Jacobson (2016).
4.1 DETAIL REPRESENTATION.
We compare our approaches with 5 baseline methods. 1) FFN Tancik et al. 2020) with SOFTPLUS activation and 8 frequency bands progressively trained from coarse-to-fin, where a total of 8 hidden layers each of size 256 are used to match our model size; additionally we apply skip-connection in the middle layer as proposed in DeepSDF Park et al. (2019). 2) NGLOD (Takikawa et al. 2021) trained using 4 and 6 levels of detail (LODs) corresponding to 643 and 256 spatial resolution spectively, with LOD4 comparable with our model in terms of the number of parameters. 3) bascline SIREN, for which we trained three different variations in hope of overcoming training divergence is- sues;. 4) direct residual, where we compose the signed distance value simply as the sum of base and displacement nets, i.e. f (x) = Afw (x) + A/wD (x). 5) D-SDF (inspired by Pumarola et al. 2021 Park et al. (020aa), which represents the displacement
T vector, i.e. f = f(x + A). where A E R is predicted in the second network. We follow network specs of D-Nerf Pumarola et al. 2020), which contains two 8-layer MLP networks with RELU activation and positional encodings. Among these, NGLOD, direct residual and D-SDF requires ground truth SDF for supervision, the rest trained using training loss. Two-way point-to-poin distance and normal cosine distance are computed as the evaluation metrics on 5 million points randomly sampled from meshes extracted using marching cubes with 5123 resolution.
7

---

(with
0.02)
with
0.05)
0.01
0.02
0.05
0.1
0.2
0.01
0.02
0.05
0.1
0.2
point-to-poin distance ( -10 -10-3)
1.178
1.171
1.147
1.146
1.149
1.146
1.147
1.147
1.149
1.152
normal cosine distance (-10-%
1.525
1.490
1.252
1.251
1.260 ! 1.254
1.253
1.251
1.250
1.274
Table 3: Study of the hyperparameters a (left) and (left). The reconstruction accuracy remains stable and highly competetive throughout hyperparameteess variation.
As shown in Table and Figure 7 our method outperforms the baseline methods with much higher reconstruction fidelity. NGLOD with 6 LODs is the only method onpar with ours in terms of de- tail representation, however it requires storing more than 300 times as many as parameters as our model. SIREN networks with larger w have severe convergence issues even with sphere initialization (c.f. Implementation Details) and gradient clipping. Direct residual doesn' 't enforce displacement directions and produces large structural artifacts. D-SDF yields qualitatively poor results, as the dis- placement net is unable to learn meaningful information (more analysis is shown in section B.1.2
4.2 ABLATION STUDY
We study the contributions of different design components, namely the displacement scaling a tanh, the attenuation function X and the progressive training.
As Table 2 shows, all the test modes converge within comparable range, even for the model with the least constraints. This shows that our model is robust against violations of theoretical assumptions specified in See 3.1 At the same time, the performance rises with increasingly constrained architecture and progres- sive training, suggesting that the proposed mechanisms further boost training stability.
or tanh
prog. training
average CD -10
1.44
1.41
1.38
1.24
Table 2:
Ablation study. Our model benefits from the proposed architectural and training designs, yet it is also robust against variations.
Table shows that in reasonable range of and there is very little variance reconstruction quality, in- dicating the robustness w.r.f. parameter selection. If ar is too small, the displacement may no longer be sufficient to correct the difference between the base and detailed surface, causing the slight in- crease of chamfer distances in the table for {0.01, 0.02). When v is too large (0.2), i.e. the high frequency signal is not suppressed in the void region, which leads to higher chamfer distances due to off-surface high-frequency noise.
4.3 TRANSFERABILITY
We apply our method to detail transfer in order to validate the transferability of IDF. Specifically, we want to transfer the displacements learned for source shape different aligned target shape. In the first test scenario, the base shape is provided and lies closely to the ground truth detailed surface. In the second scenario, we are only provided with the detailed shapes and thus need to estimate the base and the displacements jointly. The pipeline consists of the following steps: 1) train W by fitting the source shape (or the source base shape if provided), 2) train TWD, M and the query feature extractor ó jointly by fitting the source shape using equation S while keeping A fixed, 3) train Awa by fitting the target shape (or the target base shape if provided), 4) evaluate equation 5 by replacing Nwo with Awwg. To prevent the base network from learning high-frequency details when the base unknown, we use wn = 5 and three 96-channel hidden layers for Aw2.
Example outputs for both scenarios are shown in Figure TO where the base shapes are provided for the shorts model. We use 323 and 1282 grid (for the frontal view), for the shorts and face model respectively in é extract the query features. Our displacement fields, learned solely from the source shape, generate plausible details on the unseen target shape. The transferred details contain high-freuuency signals (e.g- the eyebrows on the face), which is challenging for explicit representations. However, for the second scenario the performance degenerates slightly since the displacement field has to compensate errors stemming specifically from the base SDF.
In additional, we evaluate the design of the transferable IDF model by removing the mapping net and the convolutional context descriptor Φ. For the former case, we drop the FiLM conditioning
8

---

source (top)/ target (hottom)
w/o mapping
purs wlo
DGTS
Figure 9: Transforring spatially- geometric details using various methods. Small to severe distortions are introduced when removing different components of the proposed transferable IDF. Thanks to the combination of global/local query feature, method transfers spatially-variant details while Herrz et al. (2020) only handle spatially-invariant isometric details.
source
learned source basc
leurned buse displacement
source
Jearned source base
learned
displacement
target
larget base learned turget trunsferree displacement
target
learned target base
Larget base transferred displacement
Figure 10: Transferable IDF applied to detail transfer. Left: the base shape is provided and lies closely to the ground truth detailed surface; right: only the detailed shapes provided, thus the base and the displacements need to estimated jointly.
and simply use concatenation of & (x) and f (x) as the inputs to Awo; for the latter we directly use the normal at the sampled position as the context descriptor, i.e. e (x) = V (x). As Figure shows, the removal of mapping net and o lead to different degrees of feature distortions. We also compare with the DGTS (Hertz et al. 2020), which fails completely at this example since only consumes local intrinsic features. Furthermore, the effect of scaling is shown in Figure g where using unscaled f as input to TwD leads to artifacts at the boundary.
CONCLUSION AND LIMITATIONS
In this paper, we proposed new parameterization of neural implicit func- tions for detailed geometry representation. Extending displacement mapping, a classic shape modeling technique, our formulation represents a given shape by smooth base surface and high-frequency displacement field that offsets the base surface along its normal directions. This resulting frequency parti- tion enables the network concentrate regions with rich geometric details, significantly boosting its representational power. Thanks to the theoretically grounded network design, the high-frequency signal is well constrained, and as result our model shows convergence qualities compared to other models leveraging high-frequency signals, such SIREN and positional encoding. Furthermore, emulating the deformationninnvaiine quality of classic displace- ment mapping, we extend method enable transferability of the implicit displacements, thus making it possible to use implicit representations for new geometric modeling tasks.
Figure 8: Detail trans- for without scaling f.
limitation our detail transfer application is the necessity to pre-align the two shapes. future work, we consider exploring sparse correspondences part of the input, which is a common practice computer graphics applications, to facilitate subsequent automatic shape alignment.
ACKNOWLEDGMENTS
This work is sponsored by Apple's AI/M PhD fellowship program.

---

REFERENCES
Sketchfab. https://sketchtee com 2021.
Stanford 3d scan
2021.
Ronen Basri, Meirav Galun, Amnon Geifiman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In International Conference Machine Learning, pp. 685-694, PMLR, 2020.
Sema Berkiten, Macicj Halber, Justin Solomon, Chongyang Ma, Hao Li, and Szymon Rusinkiewicz. Learning detail transfer based on geometric features. Computer Graphics Forum, 36(2):361-373, 2017.
Henning Biermann, Ioana Martin, Fausto Bernardini, and Denis Zorin. Cut-and-paste editing of multiresolution surfaces. ACM Transactions on Graphics (TOG), 21(3):31--321, 2002.
Mario Botsch, Leif Kobbelt, Mark Pauly, Pierre Alliez, and Bruno Lévy. Polygon mesh processing. CRC press, 2010.
Rohan Chabra, Jan E Lenssen. Eddy 1lg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruc- tion. In European Conference on Computer Vision, pp. 608 625. Springer, Springer International Publishing, 2020.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Pe- riodic implicit generative adversarial networks for 3d-aware image synthesis. arXiv preprint arXiv:20/2.00922 2020.
Xiaobai Chen, Tom Funkhouser, Dan B Goldman, and Eli Shechtman. Non-parametric texture transfer using meshmatch. Technical Report Technical Report 2012-2, 2012.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. arXiv preprint 1rXiv:2012.09161, 2020a.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 5939-5948, 2019.
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 45-54, 2020b.
Zhiqin Chen, Vladimir Kim, Matthew Fisher, Noam Aigerman, Hao Zhang, and Siddhartha Chaud- huri. DecorGAN: 3d shape detailization by conditional refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Robert I. Cook. Shade trees. In Proceedings of the 11th annual conference on Computer graphics and interactive techniques, pp. 223-231, 1984.
Robert L Cook, Loren Carpenter, and Edwin Catmull. The reyes image rendering architecture. ACM SIGGRAPH Computer Graphics, 21(4):95-102 1987.
Boyang Deng, John P Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. Nasa: neural articulated shape approximation. arXiv preprint arXiv:1912.03207, 2019.
Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cyxnet: Learnable convex decomposition. In Proceedings of the IEEF/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 31-44, 2020.
Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 3(7)e111, 2018.
10

---

Sarah F Frisken, Ronald N Perry, Alyn P Rockwood, and Thouis R Jones. Adaptively sampled distance fields: A general representation of shape for computer graphics. In Proceedings of the 27th annual conference Computer graphics and interactive techniques, pp. 249-254, 2000.
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7154-7164, 2019.
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4857-4866 2020.
Zekun Hao, Hadar Averbuch-Floy Noah Snavely, and Serge Belongie. Dualsdf: Semantic shape manipulation using two-level representation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 7631-7641, 2020.
Amir Hertz, Rana Hanocka, Raja Giryes, and Daniel Cohen-Or. Deep geometric texture synthesis. ACM Trans. Graph., 39), July 2020. ISSN 0730-0301. doi: 00.1145/388666933322477 URL
https://doi. www.ooorr.g..........eeo.
Amir Hertz, Or Perel, Raja Giryes, Olga Sorkine- Hornung,, and Daniel Cohen-Or. Progressive encoding for neural optimization. arXiv preprint arXiv:2104.00 104.0014.,, 2021.
Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang. Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6001-6010, 2020.
Manyi Li and Hao Zhang. D²im-net: Learning detail disentangled implicit fields from single images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 15631-15663. Curran Asso- ciates, Inc., 2020. URL httpps://proceediingg b4b758962f 7808 74 777888877766b882288888 (ateebbb822ffffaaa -awaaaab---appeeppee pdf
neurips wwtpp//ppe//22////ie.....
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 160iv: 03983, 2016.
Julien NP Martel, David Lindell, Connor 7. Lin, Eric Chan, Marco Monteiro, and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. arXiv preprint arXiv:2105.02788, 2021.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc-
cupancy networks:
Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4460-4470, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. European Conference Computer Vision, pp. 405-421. Springer, Springer International Publishing. 2020.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 3504-3515, 2020.
Yutaka Ohtake, Alexander Belyaev, Mare Alexa, Greg Turk, and Hans-Peter Seidel. Multi-level partition of unity implicits. ACM Trans. Graph., 22(3):463 470, 2003, ISSN 0730-0301. doi: 10.11445882262.82222 URL https:/t org 15/ 882262.222222 02222222..22993 882293.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2019.
11

---

Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Brutin-ruualla.. Deformable neural radiance fields. arXiv preprint arXiv:2011.12248, 2020a.
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo arXiv:201 I. 1.19948, 2020b.
Deformable neural radiance fields. arXiv preprint
Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In European Conference on Computer Vision (ECCV), Cham, August 2020. Springer International Publishing.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI Conference Artificial Intelligence, 2018.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer D-nerf: Neural radiance fields for dynamic scenes. arXiv preprins 1eXiv:2011.1390 2020.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguee D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- ence on Machine Learning, pp. 5301-5310. PMLR, 2019.
Shunsuke Saito. Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. Proceed- ings of the IEEE/CVF International Conference Computer Vision, pp. 2304 2314, 2019.
Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned
implicit function for high-resolution 3d human digitization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 84-93, 2020.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-awart neural scene representations. arXiv preprint 1906.0/:99000661 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 7462-7473. Curran Associates, /cc//ppap//22///llee/
Inc., 2020. URL https: htttps::/pproceedingg neereeeeidiiinnnnenis..es....eer 53c04118df 12c1 3ac3 Paper. .ddf
Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. arXiv preprint arXiv:201 1.026 2020.
Olga Sorkine and Mario Botsch. Tutorial: Interactive shape modeling and deformation. In EURO- GRAPHICS, 2009.
Olga Sorkine, Daniel Cohen-Or, Or, Yaron Lipman, Mare Alexa, Christian Rössl, and H-P Seidel. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH sympo- sium on Geometry processing, pp. 175-184, 2004.
Kenshi Takayama, Ryan Schmidt, Karan Singh, Takeo Igarashi, Tamy Boubekeur, and Olga Sorkine. Geobrush: Interactive mesh geometry cloning. Computer Graphics Forum, 30(2):613 (22-222,,, 2011 1.
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometrie level of detail: Real-tine time rendering with implicit 3D shapes. In Proceedings of the IEEÉ/CVF Conference Computer Vision and Pattern Recognition, 2021.
12

---

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 7537-7547. Curran Associates, Inc., 2020. URL https:: ngs neurips. .appepppaaaeiccceeppp.ee...
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Carsten Stoll, and Christian Theobalt. Patchnets: Patch-based generalizable deep implicit 3d shape representations. In Euro- pean Conference on Computer Vision, pp. 393-309. Springer, Springer International Publishing, 2020.
Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In International Conference on Neural Information Processing, pp. 264-274. Springer, 2019.
Zhiqin John Xu. Understanding training and generalization in deep learning by fourier analysis. arXiv preprint arXiv: arXiv:/808.00 04295, 2018.
Haotian Yang, Hao Zhu, Yanru Wang. Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao. Facescape: large-scale high quality 3d face dataset and detailed riggable 3d face prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lip- man. Multiview neural surface reconstruction by disentangling geometry and appearance. Ad- vances Neural Information Processing Systems, 33, 2020.
Wang Yifan, Noam Aigerman. Vladimir G Kim, Siddhartha Chaudhuri, and Olga Dorkine-Homung, Neural cages for detail- ppeeserpresr 3d deformations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 75-83, 2020.
Lexing Ying, Aaron Hertzmann, Henning Biermann, and Denis Zorin. Texture and shape synthesis on surfaces. In Eurographics Workshop on Rendering Techniques, pp. 301-312. Springer, 2001.
Howard Zhou, Jie Sun, Greg Turk, and James M Rehg. Terrain synthesis from digital elevation models. IEEE transactions on visualization and computer graphics. 13(4):834-848, 2007.
Kun Zhou, Xin Huang, Xi Wang, Yiying Tong, Mathieu Desbrun, Baining Guo, and Heung- Yeung Shum.
Mesh quilting for geometrie texture synthesis. In ACM SIGGRAPH 2006 Papers, SIG- GRAPH '06, pp. 690-697, New York, NY, USA, 2006. Association for Computing Machinery. doi: 10. 1791522./79922221144494 URL htttt:://d/dd ottpeii.rrg11...11/1111 http:tpp.s//all.or///.....1..1
ISBN 1595933646.
1179352. 1141942
Qingnan Zhou and Alec Jacobson. Thingi10k: dataset of 10,000 3d-printing models. arXiv preprint arXiv:1605.047977 2016.
A PROOFS
Theorem 1. If function f f: RR. + IR is differentiable, Lipschitz-continuune with constant L and Lipschitz-smooth with constant M, then (V V (x) L8.
Proof. If a differentiable function f is Lipschite-coontiuouss with constant and Lipschitz-smooth with constant M, then
V (y) & M | yll
(7)
(V/0x)) - (/) V/
by equation
< 18
by equation 6
13

---

Corollary 1. If signed distance function f satisfying the eikonal equation up > 0, ||v.|| - < is Lipschitz-mmotth with constant M, then V (x + x V
(1++)]6)M.
Proof. / 1
<e 3 |V < < This means is Lipschitt-coontimous with constant
e+ 1. Then by Theorem
f X + 8V (xx) Vf (x) [6](1 e)M.
Finally we show the upper bound for the normalized gradient, i.e.,
ñ n n|| - + c [8] M,
(8)
where n
Vf(x) 4 [V][]]]]
and - x d (x) n with (x) denoting the small displacement.
Proof.
V (x)
Vf x
ñ n||
(9)
[ [x)]
V [x)
For brevity, we denote V and V (x) as u and v. Without loss of generality, we assume
||||| s |/]l. Then
(10)
(11)
v|l
by Eikonal constraint
(12)
/V (x) f (x)]
(13)
d ((x)
( -
4
x
[V]
(14)
d(x)
Since (x)) small and [ (x)l close 1, we can set
Thereby using
|V (x)ll
Corollary we conclude
1
V x d
( [V](x)]
(x)))
1 18|M,
1
(15)
thus
| n|| - 1+c
I-c 8|..
(16)
Eq. equation I1 can be proved follows
u
(17)
[v]
which depicts the distance of the unit sphere projections of u and v. Obviously, as shown in |d]
Figure II
< ||d'll if <d,, e) > 90".
Since e ||ll M v and | M 2 0), to show that <d,, e) 2 90 is the same as to show
that < <d. 90" Indeed:
14

---

Tu
VI
Figure 11: Sketch for proof.
(d.v) v)
(18)
|
M
||v||
(19)
||]
by Cauchy- inequality
(20)
(21)
B ADDITIONAL EXPERIMENTS AND RESULTS.
B. ADDITIONAL INFORMATION TO THE COMPARISON IN SEC 4.1
B.1.1 ADDITIONAL EVALUATION RESULTS
Below we show the per-model Chamfer distances in addition to the average shown in Table I More qualitative results are shown in Figure 16
points to point
3
-10
model
Progressive FFN
NGLOD (LOD4)
NGLOD (LOD6)
SIREN-3 60
SIREN-7
SIREN-7
Direct Residual
D.SDF
Ours
30
60
angel
6.00/4.19
2.28/2.87
1.47/1.43
251/87.9
1.30/0899
asian dragon
4.96/4.22
1.6614.05
1.0311911
6.15/5.25 28
3.65/2.36
7.24/4033
269/92.7
2.84/1.80
0.93/1.66
camera
4.62/1.51
1556/1.15
1.32/0.62
6.50/2.38
4.11/1.10
281/38.5
1.99/2.01
compressor
5.55/1.35
1.64/0.88
1.52/0.44
8.83/2.82
4.63/0.82
+
330/56.9
2.66/3711
1.39/0.23
dragon
5.1044.00
1.803.77
1.39/2.37
7,044475
3.8022.49
263/76.4
2.68/1.21
1.24/1.50
dragon warrior
5.94/7.25
2.46/8. 12
1.52/5.21
7.27/8.45
3.68/4.83
5.96/801
6.09/9.77
2.22/4.46
1.47/4.66
dragon wing
5.68/5541
2,01/4.98
1.772.87
6.4055.46
7.92/3.84
+
167/76.7
2.66/4.09
1.31/1.49
dragon china
6.20/2.31
dragon
4.39/3.13
1.833.57
1.24/1.17
7.67/4.69
5.50V2.
J
173/86-9
3.21/4993
1.10/0.5
helmet
4.79/1.02
1.70/08771
1,40/0.400
8.40/2.72 72
5.19/0.83
263/94.6
2.59/1.99 .9
1.29/0.13
hunter
4.17/4.66
2.03/5.08
1,18/2.08
8.96/6.66
3.40/2.58
3.39/3.64
2.61/4.89
0.91/1.14
luyu
7.22/4.29
2.19/3.30
1.53/1.11
8.98/6.83
6.1663.16
206/94.6
5.10/9.76
1.28/1.02
dmgon
7.48/5.97
2.3716.
1.9/2.67
10.1/9.56
5.05/4.07
66.1 149,8
3.26/6.24
1.30/1.43
ramesses
4.24/2.30
1.47/2.47
0.97/1.93
6.40/3.77
4.20V2.33
+/-
3.78/6.60
3.16/9.66
0.92/1.58
1.304.77
6.27/7.48
1,0/2.92
Vase Lion
5.92/1.93
1,86/2. 18
1. 7.39/0.77
39/9/25.6
4.68/1 02
+.
227/54.8
2.26/2.31
1.310.433
AVG
5.4733777
2.27/4.24
1.35/1.97
9.85/6.64
4.85/2.56
-16.02
181/59.0
2.85/4.39
1.22/1.25
Table 4: Detailed quantitative evaluation (corresponding to Table 17
B.1.2 ANALYSIS
We provide further diagnosis for the underwhelming results from the direct residual and the D-SDF models. Direct residual composes the final SDF simple sum of the base SDF and residual value. We train this model also with the attenuation and scaled tanh activation, and supervise both the base SDF and the composed SDF to stablize the base prediction. However, as shown in Figure 12h, the composed SDF often contain large structural errors. Further inspections show that during the training, such structural errors change quickly and the reconstruction oscillates in scale.
15

---

direct residual
b. D-SDF
Figure 12: Examples of the direct residual and D-SDF models.
Chamfer distance
points point distance / normal cosine distance -10-2
poisson reconstruction
training points
noise
40000
0.002
1.07/7.54
1.08/7.78
40000
0.005
1.05/7.57
1.08/7.82
400000
0,002
1.00/6.01
1.04/6.63
400000
0.005
1.00/5.99
1.04/6.60
Table Quantitative evaluation given sparse and noisy inputs.
These indicate that the without the directional constraints on the displacement, the two networks are sufficiently coupled and interfere with each other during training.
In D-SDF the displacement not enforced to be along the normal direction. D-SDF yields compet- itive quantitative result, but Figure 12p shows, the predicted displacement vectors are homoge- neous, indicating that due to the lack of constraints the displacement network is not incentivised to return meaningful outputs.
B.2 DISCUSSION ABOUT w/ AND WD
As pointed out in Section 3.2 w controls the upper bound of the frequency the network is capable of representing. When using a single STREN, larger w can represent higher frequencies (more details) but at the same time tends to create high-frequency artifacts and issues with convergence. Therefore, the choice of Wg and wp is guided solely by two simple criteria: 1) wB should be relatively small to provide smooth and stably trainable base surface. wD should be sufficiently large represent the amount of details observed in the given shape.
Based on the empirical experience of the previous work Sitzmann et al. (2020), for baseline STREN network, w = 30 provides a good balance for stability and detail representation. Based on this value, we choose wB 15, that the base is smoother than the input surface, thereby creating necessity for the displacement field; wp = 60 is chosen empirically as a value that is capable of representing the high-frequency signals exhibited in the high-resolutions shapes input shape is very simple and smooth (e.g. the shape in the first row of Figure 13), the base SDF with wi 15 already sufficient represent the groundtruth surface, and the displacement has little impact. In order to enforce frequency separation as in the detail transfer application, one can as shown in Figure (3))).. For very detailed surfaces. wD needs be high enough to enable sufficient resolution of the displacement field. We choose wp = 60, which is suitable for all the tested shapes. When keeping up fixed, varying wB determines the smoothness of the base, therefore also decides how much correction the displacement network must deliver. If wB is too small, the displacement network can become overburdened with the task, leading to faulty reconstruction and training instabilities.
tested. If the
reduce WB (e.g.
B.3 STRESS TESTS
While we used dense and clean sampled point clouds as inputs in the paper, as our focus is on detail representation, examine the behavior of our method under noisy and sparse inputs. Specifically, we train our network with 400 thousand and 40 thousand sampled points (10% and 1 of the amount in our main experiment, respectively), and added 0.002 and 0.005 Gaussian noise both
16

---

buse
compooeed
base
composed
groundtruth
(4) W F
15, WD
(b) WB
5, W
15 composed
compowed
groundtruth
(c) WB = 30, WD = 60
(d) 5, WD 60
Figure 13: Effect of wa and wp demonstrated on meshes with different level of details. For smooth low- resolution meshes such as example shown in the row, a small suffices to represent all the details the given mesh. In this case, the base SDF (e.g. wy = 15) alone can accurately express the surface geometry, rendering the displacement network unnecessary, as result the composed surface is indistiggussabble from the base surface, shown in (a). To enforce detail separation, On the other hand, given detailed meshes (such to capture the high-frequency signals. When keeping wp fixed, reducing wp increases the smoothness of base SDF, as shown in (c) and (d), therefore also increases how much correction the displacement network 0, we would have single high-frequency SIREN, which is subject to
reduce wB, e.g. the second row), wD ought be large enough
shown in (b). able
must deliver. In the extreme case, convergence issues, shown in Tab 4
400K crriiiinggoinnt
40K training
input
ouTS
Paisson
Input
ours
recms.
noise 0.002
ground truth
nosee 0.005
Figure 14: Qualitative evaluation given sparse noisy inputs.
the point normals and the point positions. From the qualitative and quantitative results shown in that method recovers geometric details better than Poisson
Figure 14 and Table we reconstruction given sufficient training data (c.f. the left half of the figure). When the training sample is sparse, our method tends generate more high-frequency noise result of overfitting.
17

---

targets
learned source base
learned target base
learnod source
largels with detail from the sourcee
Figure 15: Detail transfer to multiple similarly aligned target shapes. Given a (detailed) source shape, we train base network 8 represent the smooth base surface (see learned source baxe), as well feature extractor & and a transferrable displacement network T to represent the surface details. During detail transfer, we only need to fit the lightweight base network for each new target, while the feature extractor and displacement net can be applied to the new shapes without adaptation.
B.4 DETAIL TRANSFER.
The proposed transferable IDF makes it possible to deploy and TwD a new base surface without any fine-tuning or adaptation. In other words, to transfer details to multiple targets, we only need to fit new base network Wawg The composed SDF f (x) can be computed simply by replacing W with V We show the result of talli-ttrrgett detail transfer in Figure [5] worth mentioning that even though the point extractor is trained on single source shape, it is able to generalize across different identities thanks the built-in scale and translation invariance.
B.5 INFERENCE AND TRAINING TIME
Training the models as described in the paper takes 2412 seconds (around 40 minutes), which amounts to 120 epochs, i.e., around 20 seconds per epoch, where each epoch comprises 4 million surface samples and million off-surface samples. comparison, the original implementation of NGLOD6 takes 110 minutes to train 250 epochs, where each epoch comprises 200000 surface sam- ples and 300000 off-surface samples. As for inference, using the same evaluation setup. NGLOD6 takes 193.9s for 5123 points, while our inference takes 250.06 seconds for 5123 query points. All benchmarking is performed on a single Nvidia 2080 RTX GPU. These timings could be improved by optimizing the model for performance, which we did not. For example, instead of using autodiff for computing the base surface normals, one could exploit the fact that the differentiation of SIREN also SIREN and explicitly construct computation graph for computing the base surface normals.
18

---

ITN
SIREN-3
SIREN-7
direct reisdual
D.SDF
60
30
Comme
NGLOD LOD4
NGLOD LOD6
OUTS
ground truch
FPN
SIKEN-3
SIREN-7 30
direct reisdual
D-SDF
60
NGLOD
NGLOD
OuTS
ground truth
IFN
SIREN-3
SIREN-7
direct reisdual
D.SDF
60
30
OnTs
ground
LOD4
LOD6
Figure 16: Comparison of detail reconstruction (better viewed with zoom-in). Methods that did converge omitted in the visual comparison.
19

---

ITN
STREN-3 60
SIREN-7
direct reisdual
D.SDF
30
OurS
ground
LOD4
LOD6
FFN
SIREN-3
SIREN-7
direct reisdual
D-SDF
ours
ground
LOD4
LOD6
FEN
SIREN-3
SIREN-7
direct reistual
D.SDF
ours
ground
LOD4
LOD6
Figure 16: (Cont.) Comparison of detail reconstruction (better viewed with zoom-in), Methods that did not converge are omitted in the visual comparison.
20


Reviews:
[{"review_id": "fvIcDvlaaxP", "reviewer": "Reviewer_WHEF", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "he paper is well written, makes its contribution very clear and gives a distinguished overview about related work. I had no problems following the text, and the equations are \n\nThe papers main contribution is the introduction of a two-level implicit representation (both based on SIREN) for 3D geometry which works exceptionally well. The general idea was already introduced for single-view reconstruction (in Li & Zhang as mentioned by the authors), but this papers evolves the setting to work for general geometry representation, including an inverse displacement and making the displacement transferrable by using a kind of feature-based UV map. \n\nWhile the existing two SIREN network setup lowers the value of the contribution, the elegant solution for some geometry representation specific problems. These solutions are the expensive comparison to the ground-truth which is solved by including an inverse implicit field, and learning a feature based UV field (computing consistent UV maps on different surfaces is very hard). I could see especially the second part being reused in other applications as they allow to transfer the UV to similar shapes. \n\nThe results are visually impressive, especially the texture transfer, closer to the ground-truth than all competitors all while using a smaller model size. The results are both smooth in areas where this is correct, as well as able to show details where necessary.  This could be incredible useful for applications based on implicit 3D representations that struggle with training complexity of highly detailed models. \n\nThe ablation study is well done, however the quantitative comparison is only done on a single dataset, SketchFab-16. I do not know this database and the provided link only goes to a general website with a wide collection of 3D models (some of them not free). This makes it impossible to exactly reproduce the results. Additionally, it seems all models on this page are of a certain high quality, it would have been interesting to at least see some other examples with shapes of different quality. \n\nMinor comments:\n- I find that the practice of the listed citations without brackets around them very distracting when reading. I do not know if this is part of the conference style guide but I would recommend to change it. \n- Figure 4: It would be great to write the w parameter directly under/above each image. ", "summary_of_the_paper": "The paper proposes a an implicit representation of surface displacement fields that is well learnable by neural networks. The network is based on two SIREN architectures that separate low and high frequency information in the model. One returns an SDF of the rough shape while the second returns a displacement field at the surface of the SDF. In addition to this architecure (which existed before but for a different application), the paper introduces an inverse implicit mapping that speeds up the evaluation of the loss, and a feature based UV map that can be transferred between similar shapes. The paper compares to previous work on SketchFab, and specifically evaluates against a vanilla SIREN model where this method results in both a smaller model size as well as more detailed reconstruction. Additionally, there are experiments that show how details can be transferred across different (but similar) models. ", "summary_of_the_review": "While neither the SIREN architecutre nor the displacement field is a novel contribution, their combination is not trivial and shows exceptional results in terms of quality and model size. In addition, the paper presents some elegant solutions for efficiently evaluating the loss in this new setting, and using a feature based UV mapping that can be transferred to similar shapes. \nI think these are sufficient for publication. The quantitative evaluation is a bit lacking, and the final version of this paper should include an exact list of shapes that were used in the evaluation. op", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "J8KpbxF8JA_", "reviewer": "Reviewer_G54X", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nS1: The paper presents an interesting idea. Although this is a straightforward idea, using implicit displacement fields is up to my knowledge new.\n\nS2: Related work is OK, and the paper is well organized.\n\nS3: Several experimental results are presented, including ablation studies. However, essential comparisons are missing (see the weaknesses.)\n\nWeaknesses:\n\nW2: The method is mostly constructed on top of previous methods; there are no network changes or losses. There is a contribution in the signed distance function and a pipeline for transferable implicit displacement fields. Why are we using two SIRENs for f and d? Shouldn't the d be a simpler network?\n\nW3: Considering the experimental result. I feel that the method will fail with noise because of things like the need for normal to the points. This is a very relevant fact, and it is not described in the document. There are small tests with noise in the appendices, but the level of noise added is almost 0.", "summary_of_the_paper": "The paper addresses the problem of obtaining a 3D surface reconstruction from point clouds. This has been one of the most studied topics in 3D vision, with different kinds of approaches. The authors tackle this problem using implicit representations, specifically using signal distance functions and query features. The main contributions are a consideration of implicit displacements for modeling different frequency levels and introducing a transferable implicit displacement field that replaces the common coordinates inputs with constructed transferable features. ", "summary_of_the_review": "Although the proposed method builds mainly on top of previous techniques, the paper presents some interesting contributions. However, it is not clear how this new representation will work with noise, and I believe it would be beneficial to clarify this. However, if the other reviewers think the paper is ready to be accepted, I will not be against it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DtgeigL0E99", "reviewer": "Reviewer_GmTU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The paper is well-written and well-motivated. The authors have done adequate derivation from the classic explicit displacement mapping method, making the proposed method theoretically sound.\n- The idea of implicitly representing both base surface and displacement map with neural networks is new and elegant. Transferable implicit displacement field (IDF) also adds to the novelty of this framework.\n- The proposed method outperforms all baseline methods both quantitatively and qualitatively. The authors have done adequate ablation studies to validate the method design. The transfer experiments are impressive and show great potential in applying the method to downstream shape processing tasks such as detail modeling.\n\nWeakness\n- It would be helpful to explain how some of the hyper-parameters are chosen e.g. attenuation factors $\\nu$, and frequency factors $\\omega_B, \\omega_D$\n\nOther\n- Is it possible to perform hierarchical displacement mapping / shape modeling based on this idea? Any discussion is welcomed.\n\n", "summary_of_the_paper": "This paper proposes an implicit representation for shapes, which utilizes two SIREN networks of different frequencies representing base surface and displacement respectively. In addition, the authors introduces transferable implicit displacement fields, which enables the transferability of displacements between aligned shapes. The method is a natural extension to the displacement mapping method and is theoretically grounded. The evaluation results show that this method could represent shapes with sharper details while using less storage compared to baseline methods.", "summary_of_the_review": "The paper proposed a new method for implicit shape representation. The authors have done adequate experiments to show the effectiveness and transferability of the proposed method. However the method is an intuitive combination of traditional displacement mapping method and SIREN, lowering the novelty of this method slightly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9qkBvDc-GNg", "reviewer": "Reviewer_Eb6T", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n(1) To represent both the coarse surfaces and displacements implicitly by two neural networks is new. The results show that the proposed method can keep detailed 3D geometry while remaining a reasonably small network size.\n\n(2) Instead of naively building the displacement SDF as residuals, the proposed displacement field and attenuation function (eq(1),eq(3)) are interesting. Ablation studies also demonstrate their effectiveness.\n\nWeaknesses\n\n(1) I would like to see more experiments and discussions about how the parameters ωB and ωD (ωB = 15 and ωD = 60) are chosen in the paper. How would they affect the performance in different objects, from simple to complex surfaces? To me, it seems that these two parameters would be object-dependent. \n\n(2) Is this method generalizable to other baseline models (other than SIRENs)? ", "summary_of_the_paper": "This paper proposes an implicit neural representation for 3D geometry, named implicit displacement fields. The method is consists of two SIREN networks as baseline models to approximate: a coarse base shape representation (low-frequency) and an implicit displacement field (for high-frequency details). Different from previous works, the displacement field is implicitly represented by a SIREN network. The coarse and fine networks design also enables applications such as shape manipulation.", "summary_of_the_review": "The proposed implicit displacement fields have the ability to represent detailed geometries in relatively small network sizes. Experiments are done thoroughly to demonstrate the effectiveness of each component of the method.  If shapes are pre-aligned properly, the implicit displacement also enables the shape manipulation application. Despite having some uncertainties (see weaknesses), the reviewer holds a positive opinion on this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a normal map-inspired implicit surface representation involving a smooth surface whose high frequency detail comes from normal displacements.  Reviewers were impressed with the results and theoretical discussion in the paper.  The AC agrees.

The authors were responsive to reviewer feedback and addressed some questions about parameter choice during the rebuttal phase, including new experiments/discussion in the supplementary document.  Note the response to reviewer WHEF notes that the authors will be releasing data/code; the AC strongly hopes the authors are true to their word in that regard.

The AC chose to disregard some comments from reviewer G54X regarding tests with noise, as this method appears to be tuned to computer graphics applications; the level of empirical work here aligns with past work in the area.  Of course the authors are encouraged to include some tests responding to the reviewer comments in the camera ready.  The AC also found the score from reviewer WHEF to be somewhat uncalibrated with the tone of their review, but of course their assessment is quite positive nonetheless.


One small comment:  The abstract appears a bit strangely on the OpenReview site because of line breaks; if possible, please remove the line breaks.

Another small comment:  The "spectral shape representation" phrase used a bit in the discussion below might not be advisable, as this phrase typically refers to the intrinsic spectrum of a shape (e.g. Laplace-Beltrami analysis)
Decision: Accept (Poster)