Geometry Processing with Neural Fields
Authors: Guandao Yang
Serge Belongie
Bharath Hariharan
Vladlen Koltun
Venue: NeurIPS 2021 Poster (2021)
Keywords: Geometric processing
Neural Implicit Fields
Implicit Representations
Implicit Fields
Shape Editing
Shape manipulation
Shape Smoothing and Sharpening

Abstract: Most existing geometry processing algorithms use meshes as the default shape representation.  Manipulating meshes, however, requires one to maintain high quality in the surface discretization.  For example, changing the topology of a mesh usually requires additional procedures such as remeshing. This paper instead proposes the use of neural fields for geometry processing. Neural fields can compactly store complicated shapes without spatial discretization.   Moreover, neural fields are infinitely differentiable, which allows them to be optimized for objectives that involve higher-order derivatives.  This raises the question: can geometry processing be done entirely using neural fields? We introduce loss functions and architectures to show that some of the most challenging geometry processing tasks, such as deformation and filtering, can be done with neural fields. Experimental results show that our methods are on par with the well-established mesh-based methods without committing to a particular surface discretization. Code is available at https://github.com/stevenygd/NFGP.

polygonal meshes to represent Geometry Processing with Neural Fields
Guandao Yang
Serge Belongie University Copenhagen
Bharath Hariharan Cornell University
Vladlen Koltun Intel Labs
Coreell University
Abstract
Most existing geometry processing algorithms use meshes as the default shape representation. Manipulating meshes, however, requires one to maintain high quality in the surface discretization. For example, changing the topology of a mesh usually requires additional procedures such as remeshing. This paper instead proposes the compactly store complicated shapes without spatial discretization. Moreover, neural fields are infinitely differentiable, which allows them to be optimized for objectives that involve higher-order derivatives. This raises the question: can geometry processing be done entirely using neural fields? We introduce loss functions and architectures to show that some of the most challenging geometry processing tasks, such deformation and filtering, can be done with neural fields. Experimental results show that our methods are on par with the well-established mesh-based methods without committing a particular surface discretization. Code is available at https:// trithuub... com/ st stevenydd//wt GP.
of neural fields for geometry processing. Neural fields
Introduction
In many graphics applications, users may want edit digital shapes using just few clicks, such as making a character bow by dragging the head downwards. Such manipulation from sparse input requires geometry processing algorithms. Most of these algorithms shapes [11]. Polygonal meshes were created initially for researchers as representation of real-world shapes that they can both interpret and manipulate [14, 52, 78). However, developing algorithms that automatically manipulate meshes is often difficult since they involve discretizing the surface. For example, changing the topology of mesh, such as turning sphere into torus, will break such discretization and require additional repair procedures such as remeshing [2. 3]. Given that shape editing is inereasingly performed by algorithms, seems worthwhile search for shape representation that is more amenable to automatic geometry processing.
An alternative the polygonal mesh is implicit representation, in which the surface is represented by a level set of field: (a) (x) c} [22, 39, 63]. Since it is easy to change shape topology using implicit representations, people have applied them for geometry processing tasks such shape merging [8, 51, 54]. These works store implicit fields using voxels or octrees, which introduce memory-intensive spatial discretization. Recent research addresses this by using continuous neural networks to represent implicit fields [19, 45, 46, 56]. These neural fields possess several advantages in addition to the merits inherited from implicit representation: they are compact to store [23, 45] and can produce high-qualis continuous surfaces at arbitrary resolutions [45, 56]. The community has achieved compelling results using neural fields in variety of applications [47, 64, 73, 84). Notwithstanding these results, it is still unclear whether shape editing tasks challenging for implicit fields (e.g., deformation and filtering) can be performed with neural fields. In this paper, we ask: can geometry processing be done entirely using neural fields?
*Email: sy466cornel1.ed. This work is done while Guandao was interning at Intel Labs.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Geometry processing tasks are challenging because they are typically under-conee constrained: user input very sparse. Therefore, we need leverage priors that characterize how a natural surface behaves. Such surface priors are usually instantiated by minimizing energy functionals that describe the physical properties of surfaces using operators from differential geometry. For example, one can encourage surfaces to be smooth by minimizing the curvature, which can be measured by the Laplace- Beltrami operator. Such operators, however, require a parameterization of the surface, which is not available for level sets of neural field. Mesh algorithms usually approximate these geometric operators using the geodesic neighbors of the surface point. This allows the algorithms to focus on intrinsic geometric properties that are independent of shape parameterization. However, it is not easy to obtain geodesic neighbors in a neural field since the surface of interest is encoded implicitly by set of points that evaluate the field function to the same value.
We posit that computing these geometric operators in neural fields requires a fundamentally different approach. The idea is to approximate the local surface of the level set using the derivatives of the underlying field. We can evaluate intrinsic geometry properties of the level set, such as curvature, using only the field derivatives [39, 54J. This allows to develop loss functions that describe surface priors such as elasticity or rigidity. To achieve this, we use the fact that neural fields are designed to be infinitely differentiable [66, 74J. The infinite differentiability of neural fields makes it possible to optimize loss functions that involve higher-order derivatives using gradient descent methods. Thus, unlike mesh-based geometry processing algorithms that approximate these objectives using surface discretizations, we can directly optimize in terms of the derivatives of the field.
Our formulation provides proof of concept that geometry processing can be done entirely using neural fields. In particular, we focus on two tasks: shape filtering (e.g., smoothing and sharpening) and deformations. These two tasks are not only essential for downstream applications but also bring out the known challenges associated with implicit representations. We first tackle shape sharpening and smoothing and show how these can be done by optimizing a loss function based on geometric properties computed via the neural field's derivatives. To achieve shape deformation, we propose to warp the neural field using a deformation vector field modeled by invertible neural network [7]. With this invertible deformation field, we derive a training objective that models the implicitly represented surfaces as elastic shells.
We compare our method with well-established mesh- baselines [70. 76). Our method can match the quality of the mesh-based counterparts without discretizing the surface. We hope that our work can inspire future generations of geometry processing algorithms using neural fields.
2 Related Work
This paper builds on two bodies of work: polygon mesh geometry processing and neural field representations. We will focus on prior work on two geometry processing tasks: shape filtering and deformation. Please refer to Botsch et al. [11] for further reading.
Mesh smoothing (and sharpening). The goal of mesh smoothing to remove figh-frequency noise and produce a smooth surface. Noise removal can be achieved by Fourier analysis [76, 79] or by modeling surface motion with a diffusion equation [26]. Smoothness can be promoted by minimizing energy functionals inspired by differential operators [50, 72, 82]. Sharpening is the inverse of smoothing. In this paper, we adopt the goal of smoothing from prior work [20, 76] and design corresponding differentiable objective for neural fields.
Implicit field smoothing. Implicit representations can be smoothed by evolving the level set according to its curvature normal [49, 54]. Many network architectures, such ReI.U MIP, bias toward smoothness when modeling an implicit field [61, 66, 74]. Regularization objectives and initialization schemes have been proposed to obtain smooth level sets from scanned surface points [5, 31]. These works mostly use smoothness as a prior when training neural fields. Our work formulates objective that allows smoothing or sharpening neural field as desired.
Mesh deformation. Mesh deformation done by either deforming the shape (i.e. vertices) [9, 69] or the space [6, 36, 41, 48, 65]. These algorithms usually draw inspiration from physics and differential geometry to develop energy functions that encourage natural deformation [10, 15, 24, 70, 71, 77]. They also linearize such objectives so that they can be solved efficiently using linear

---

solvers 19]. Recent research also applies deep learning to optimize for non non-linear deformation losses [32, 34, 38, 80, 81, 83]. Deforming a mesh, however, requires estimating the deformation objectives with spatial discretization and maintaining vertex connectivity in a way that preserves topology [42]. We deform neural fields with a continuous invertible field to cireumvent these issues.
Implicit field deformation. Prior works have studied the deformation of implicit fields for tasks like physics simulation [39, 51, 54, 55]. These works use voxels or octrees to represent implicit fields, while our paper advocates for using neural fields for deformation. There is some past work on deforming neural fields to match a target shape [12, 18, 37, 53, 58] or image [43, 57, 60) without any correspondences between the two. In contrast, we perform such deformation with very few localized user-provided correspondences, but without any other information about the target shape. Our setting very convenient for artists, but requires the algorithm to provide a strong surface prior due to the lack of densely specified target state. Other prior works editing neural fields are trained on a dataset of shapes from a particular object category [25, 29, 30, 33, 85]. Since surface priors are specified implicitly by the dataset, it is unclear how to apply these methods to
shapes. In contrast, our paper requires no training dataset and instead enforces priors inspired by more general physical properties such smoothness and clasticity. Remelli et al. [62] developed a way to differentiate iso-surface extraction, which allows deforming neural fields using objectives carried over from mesh-based algorithms. Our method avoids iso-surface extraction entirely and formulates differentiable deformation objectives directly on the implicit field.
3 Computing Surface Properties of Neural Fields
In this section, we will discuss how to compute surface properties using neural fields and establish notation for the following sections. Let an be a surface enclosing the region Qt A signed distance field (SDF) for surface a is defined as d(x) = s(x) minyean ||x yll- The sign function sxx) evaluates to 1 if x E 52 and otherwise. We will assume that a neural field approximate an SDF whose pero-isosurface represents the surface of interest. For neural field f. we denote its zero-isosurface using My. This section summarizes how to compute surface normals and curvatures from neural fields. This is well-understood and we summarize this here for convenience [39, 54, 59].
Surface normal. The surface normal is the vector perpendicular to the tangent plane and describes the local orientation of the surface. For an SDF, the outward oriented surface normal of its level set the field gradient: (x(x)) = Vx)(x). It can be shown that the SDF's gradient norm is always one: [[xd((x]]| = 1. This property can be used to ensure that the field remains a valid SDF throughout any manipulation [4, 31]. The closest point in OK from x can be found by x d(x) (x))... This property can be used to sample points from the isosurface without creating a mesh [5, 13]. Finally, since the tangent plane is perpendicular
the surface normal, we can project vectors onto the tangent plane by subtracting their projection onto the normal using the following projection matrix x: Pa(x) = I nx(x))n(x)) This projection matrix allows us characterize the tangent plane. This matrix is important for measuring how much a given deformation stretches the tangent direction [35].
Curvature. Intuitively, curvature describes how much surface deviates from a plane. This can be captured by the total derivative of the surface normal, which is also known as the shape operator: Sp(x) = (x.(x). For an SDF, the normal itself is given by the derivative of the field, thus the shape operator for an SDF is the Hessian of the field function: Ha(x) be used to capture many different notions of curvature, including mean curvature (half the trace of (x),,, denoted by R), Gaussian curvature (the determinant of Sp (x)), and principal curvatures (the eigenvalues of Sp())). We will use these to define objectives that smooth or sharpen the surface.
DPd(x), The shape operator can
4 Shape Smoothing and Sharpening
In this section, we will show that shape smoothing and sharpening can be done directly on neural fields without producing meshes. our setup, the input shape is represented as the zero-isosurface Mr of the neural field F. The algorithm needs to output neural field Go whose zero-isosurface Ma. satisfies two goals. First, MG, should preserve the global structure of M p. Second, the surface MG. should make desired changes to match the curvature of shape Ma.
3

---

Iu
[0
Smoothing and sharpening results. (A) zoomed-in input; (B) our method; (C) baseline without remeshing; (D) baseline with remeshing. We that the baseline without remeshing finds difficult to smooth or sharpen the mesh extracted from neural fields. Our method is able to achieve good results without discretizing the surface.
Figure 2:
To achieve these goals, we first instantiate Go using the same architecture and parameter values as Then optimize Go using the following objective:
/ xEU
/ xEV,
((0)
(/ 1)? dx
(IV
d (x) (x)) dx.
The first integral encourages the network Go to preserve the original shape. The second term regularizes Go to remain a valid SDF by enforcing that the norm of the gradient is (i.e., the Eikonal constraint [4, 31J. The integration is over the region U where the original neural field /' supervised over. The final term aims to smooth or sharpen the surface by increasing or decreasing the curvature ε. Setting 3 < will decrease the curvature of the output surface KG., resulting in smoother shape. B > will lead to surfaces with higher curvature details and will thus sharpen shapes. In this paper, we will use mean curvature since it's easy to compute with neural fields: Ky (x) ir (
Figure 1: L: SIREN learned fit the SDF of square; R: Zooming in on the surface.
One challenge is that the computation of curvature can be very noisy for neural fields using periodic activations. Figure 4 shows how an ostensibly smooth isosurface learned through SIREN [66] is actually quite rough when zoomed in. The curvature evaluated on such a rough surface can be too noisy be used for training. To alleviate this issue, we only compute the curvature regularization in areas where the curvature of the level set is less than certain threshold т. Formally, we define this area as V, fx U| max([ (kel) < T). We use rejection sampling to sample points from V/r when computing the loss during training.
Results. We follow prior works [20, 76) to use Armadillo [40] and a sphere with one half of
corrupted by Gaussian noise. We compare our algorithm to two baselines. The first baseline directly applies the smoothing algorithm developed by Taubin [76] to the mesh extracted from the neural field using Marching cube [44]. The second baseline applies the filtering algorithms on meshes simplified by quadratic decimation [27]. The input neural fields are created following the procedure of Park al. [56]. The results are shown in Figure The first baseline fails smooth or sharpen the surface appropriately. It only modifies the surface with high-freug -reeufrry noise. While the second baseline can filter the surface correctly, introduces discretization artifacts due to the decimation process. This suggests it is challenging to filter the neural field surface with mesh processing algorithms since these algorithms are sensitive the quality of surface discretization. On the other hand, our algorithm can produce good filtering results without the need to maintain good discretization of the surface.

---

5 Deformation
To deform a shape, the user will choose a set of deformation handles {b,) For each handles h the user will specify target location t; describing where the handle will be dragged to. Users can set t, = h, to enforce part of the surface to unchanged. Given this input, there are two goals for the shape deformation algorithm: 1) satisfying user inputs and 2) ensuring that the deformation resembles the natural behavior of real objects. The input shape will be represented by the zero-isosurface of a neural field F in our setting. The algorithm must therefore output a neural field Gg whose zero-isosurface represents the deformed shape that satisfies the aforementioned two goals.
In this setting, deformation can be formulated as solving a constrained optimization problem:
min Laf (Go, F) s.t. Lconst (Go, t,, h2) = 0, 1 sin n,
(2)
where the objective mm (Gg, F) measures how natural the deformation is. The constraints Leonst (Go, t,, h.) = 0 ensure that the result deformation matches user input.
One way to ensure natural-looking deformation is to assume that the surface behaves like thin elastic shell, which is resistant to stretching and bending [1, 70]. Traditional mesh-based algorithms model elasticity by minimizing the thin shell energy that penalizes stretching or bending happened to any local patch. Naively adapting such training objectives from mesh-based algorithms to neural fields is challenging because these algorithms assume dense correspondences between two surfaces and efficient access to geodesic neighbors.
In this section, we will address these challenges by following strategy. To measure how much bending or stretching happened to the whole surface, we will first divide the surface into infinitesimal patches and then sum the amount of bending or stretching that happened to each patch. To achieve that, we first develop a sampling schema to sample these infinitesimal patches uniformly from the input shapes (Sec 5.1). Then we warp the input shape with an invertible neural network to achieve correspondences between the input and output surfaces (Sec 5.2). Finally, we derive ways that measure the amount of bending and stretching between two infinitesimal surface patches (Sec 5.3).
5.1 Sampling
Our first step sample infinitesimal surface patches uni- formly to the surface area from the -rroossseeeecce of the neural field. Each infinitesimal surface patch can be rep- resented by surface point and its local derivatives. With this said, the problem amounts to sampling points from the zero-isosurface. Formally, we are given a neural field F that approximates the signed distance field of the input shape. The goal is to find way to sample an arbitrary number of points on its cero-isosurface MM. One way to sample points from MM is to run Langevin dynamics using the gradient of the field F [5, 13]: X(+1 = X4 (( initialized by sampling uniformly within 13. Here we also leverage an SDF property: the closest surface point to x can be computed by x F(x)F(() (Sec 3). While this process produces points guaranteed to be the zero-isosurface, the mixing time of the procedure is very long without appropriate coarse-to-fine annealing [68]. Running this process for limited iterations results in point samples that are concentrating near the high curvature area, as shown on the left side of Figure 5.
Figure 3: Sampling from dino. L: no rejection; R: ours (with rejection).
X, ~ N(x,,oI) x, are
To alleviate this issue, we instantiate the sampling procedure with x sampled from a rough shape. This can be done by first sampling uniformly from the bounded space and rejecting points that are too far away from the isosurface: Xo ~ {x E 1) < Ta). We found that т, can be set to relatively large number (e.g., about 0).. This prevents high rejection rates without sacrificing sampling quality. Figure 5.1 shows that our method creates uniform samples.
We quantitatively evaluate the effectiveness of the sampling scheme. Specifically, we compare our sampling scheme with directly applying Langevin dynamics [13] (LD Table 1). To do that, we compute the Chamfer Distance (CD) and Earth Mover Distance (EMD) between points sampled by the valuated algorithms and points sampled uniformly from the extracted mesh. For both metries, lower values indicate better performance Since larger surfaces can lead to larger CD and EMD values,
5

---

we normalize the metrics by the values obtained by comparing two sets of uniform samples from the mesh (i.e., CDr and EMDr) The results show that initializing X by rejection sampling significantly improves both CD and EMD compared to naively applying Langevin dynamics, at minor cost in sampling time.
5.2 Invertible Deformation Field
After we obtained surface points samples from zero isosur- faces of the input neural fields, the next step is to deform these surface points to the output shapes. One way to deform neural fields is to warp the coordinate space of the input network using a deformation field predicted by neural network Do [25, 43, 57]. Under this framework, the output network can be defined as (x = Fx)x
Table Our sampling method achieves better results than the baseline.
Dino
Armadillo
Metrics
LD
Ours
LD
Ours
CDr (4)
1.54
1.04
1.36
1.02
EMDr (4)
3.38
1.15
3.30
1.08
Then the constraint Leonse ( Ga, t, h) be defined Leonst (Go, t, h) = Dttt) hhl-
Time
0.15
0.21
0.12
0.18
Ideally, we would like the deformation Dg to be continuous and invertible. This will allow to create well-behaved one-to-one correspondences between points on the deform shape (i.e., x) and points on the imput shape (i.e., (x(x)). To achieve this, we make Dg to be an invertible network composed of a sequence of invertible residual blocks [7].
Invertible residual block. Prior works [7, 17] have shown that a sufficient condition for the residual block f(z) x + g(x) to be invertible is that the Lipschitz constant of function g is less than 1. The architecture of the network g(x) is usually composed of spectral normalized linear layers and Lipschitz continuous nonlinearities such as ELU [21]. The inverse of such residual block can be
computed by finding the fixed point of function y > y g(x) [7]. Intuitively, deforming using one invertible residual block amounts to moving point with deformation vector g(z)..
Lipschitz continuous positional encoding. Applying such architecture directly without modifi- cation fails to produce deformations with many different local rotations. Recent research suggests that periodic functions are essential for coordinate MLPs to predict complex signals [47, 66, 74]. Periodic functions are usually used as positional encoding activations of the form sin(ar b). where controls the frequency of the activation. To use such periodic function as part of the invertible residual block without restricting the frequency, we normalize the output of the periodic function sin sin(ax + b){a|= bringing it is Lipschitz constant below 1. Formally, the positional encoding with normalized Lipschitz constant applied one dimension
cos(2 sin sin(22xxx)
c2 sin(24
(x(x)= = /2L+1 Dig
(3)
20
20
2
2d
We will apply such encoding to each of the dimensions of the input coordinate for the body of the invertible residual block: Rx(x) = x 90 J (x), ..., Ta (x))).
Ablation. We conduct ablation studies two architec- ture choices: invertibility and positional encoding. In this experiment, we deform a neural field representing the SDF of a 2D rectangle. We optimize each ablation case satisfies the user specified constraints with following loss: 1 ""_ (h;) 2 The results are shown in Figure 4. If we replace the invertible architecture with SIREN as done in Deng et al. [25], the deformation field will tend to break topology (No inverse). Removing the positional encoding will fail to create complex field (No PE). Our architecture can create reasonable guess (No loss). This shows that our network architecture prioritizes natural deformation that preserves topology.
No inverse
No PE
Ours
n
Figure 4: Architecture ablation on 2D SDF deformation. Blue points are han- dles; Red points are targets.

---

5.3 Implicit Thin Shell Loss
At this point, we have obtained the correspondences y (x),,, with x in the output (or deformed) space and y in the input space. Our next step is to design Lafm that compares these corresponding patches to ensure natural deformation. Recall that one way ensure natural-looking deformation to minimize the amount the resistance to bending or stretching that happened during the deforma- tion [1, 70]. We will develop novel loss functions for measuring stretching and bending. Please refer to the supplement for the connection between our loss and the thin shell energy.
5.3.1 Stretching Loss
Stretching can be captured by the change of dot product in tangent space. Intuitively, local surface patch is stretched when the lengths of some tangent vectors change. The change of tangent vectors' norm can be measured by the tangent dot-product. With this said, the amount of stretch can be measured by the change of tangent dot product.
Note that do not have access to analytical surface parameterization, so need express the tangent dot product with neural fields and its derivative. We can describe the tangent dot product to the spatial dot product using the projection matrix. The tangent vector can be parameterized by projecting a vector v to the tangent space. To achieve that, we need to multiply v with the projection DG, [x) nc, (x)². where nG, (x) is the surface normal of point x. Pc, (x)v is a tangent vector in the tangent plane of x.
matrix PG. (x)
Now ready to compute the change of tangent dot-product. Let and t be two arbitrary langent vectors near point x at the deformed shape. Further assume that these vectors can be parameterized t; PG, (x)vi and t; Pa, (x) These tangent vector will be transformed by De into 1 = JD, (x)t, and t2 = Jp, (x)t,- These are tangent vectors at point at the input shape. The change of tangent dot-product with respect to these two vectors can be computed as follows:
t2 o V.P.G. (x)² (I JD, J (x)) PG, (x)v2l-
To minimize the stretch, we need to enforce that dot- dot-product stays the same for all tangent vectors. This can be done by minimize the matrix norm of PT. (I - -Jh Jb.JDo JD.) PG.. Here, we drop the function arguments x for the matrix for notation clarity. With these, we define the the stretch loss as:
C.)Go) = XEMaa
P.. (I (I-J%, JD.) Pa dx.
(5)
5.3.2 Bending Loss
Bending can be characterized by the change of surface curvature (e.g., making the surface more or less curved). Intuitively, curvature can be described as the change of tangent dot product along the surface normal direction [16, 59]. Let t = Pa. (x)vi and t2 Pc. (x)v2 be two tangent vectors at x. Consider the tangent dot product of the surface family x + /n(x). Note that this set of surface corresponding to the level sets {p]Go(p) (p) f) if Gg is approximating
SDF [54]. Then the change of dot product along the surface normal direction can be given by Hessian and directional derivative: as tf t2 t Hc, (x)t2. Intuitively, the larger this value is, the faster the surface changes when going along the surface normal direction, which means the surface has larger curvature.
d
Measuring bending amounts to measure the change of tangent dot-product derivative along the surface normal direction. Similar to the previous section, assume we have tangent vectors t; = Pc. (x)v; and t, PG.. G. (x)v,. These tangent vectors are are transformed by the Jacobian of the deformation field J J. (x) to t² = D, (x)t; and ty JD, (x)t,. Since t/ and t' are tangent vectors at point y on the surface MA. the derivative of tangent dot-product is given by the hessian matrix of field F: t/TT t22 = t² H (D)( =t (x) (Dx((x())) [J D, (x)t2. We will drop the function argument to J JD. HG.. and Pc. for notation clarity. The change of tangent dot-product derivative is:
dt (tift2 1TT2) 0 = vF' (Ha. Jb (T. Va.
(6)
If there is almost no bending happened between the infinitesimal patches around x and y, then the quantity d (tf't2 (TTt) should stay close to 0 for all pairs of tangent vectors. We will
7

---

quantify this by the matrix norm of G. (Ha. J H Pa.. Finally, we arrive at our bending loss, which minimizes the norm of the matrix that modulate the change of tangent dot-product derivatives moving along the surface normal direction:
Co(Go) xEMM,
|P (Ha. J, Hp (Dx((())) Paollr dxc
5.3.3 Computing Lg and Lo
Computing Lg and La during training requires approximating a surface integral of the form fxEMo, ((x)dx using Monte Carlo integration, which requires an efficient way to sample points uniformly from surface Ma.. While it is feasible sample uniformly from SDF, this is chal- lenging to do with Ga since there is no guarantee that Gg remains a valid SDF during the course of training. Thanks to the invertibility of Da, we can apply change of variable x D (y) to the integration, where y points the imput surface Mp:
D
SEEMOD ((x)dx
/
((x) dee (Jn, (x) (x) (y)a) (x)*) dy.
(8)
JyEMr
J D. PG. E 3x3 maps tangent vector from Mc, to Mp. We apply the extension trick from Iglesias al. [35] create matrix whose determinant equals the change of surface area. This is achieved by adding the surface normal component ne and keeping it unchanged. With this change of variable. we now can compute the losses with point samples from the input field F. Since the input field F approximates an SDF, we can use the method introduced in Section 5.1 to sample uniform points.
5.4 Optimization
Putting the losses together, we deform neural field by solving the constrained optimization problem arg min (Go) + CGo), s.t. v1 S n, [Dv(t.) h.
8
where As As are hyperparameters that determine the material properties. One way to solve this constrained optimization problem for a neural network is make the constraints soft loss adding to
the objective function: Loanst (Do) = = >n (Da(t;) - h:ll. Our final objective is: [(0) = (G.) + AgLo(Go) + XDL)
(10)
We set d to be a high value to enforce that the model satisfies user-specified input as much as possible. The user can tune A and X₂ depending on the application scenario. We optimize [(0) with Adam optimizer to obtain the output field Gg. Hyperparameters are provided in the supplement.
Deformation Results
In this section, we will demonstrate the results of our methods to shape deformation. The shapes for deformation are taken from Sorkine- and Alexa [70]. To create neural fields from these meshes, we follow the procedure of Park et al. [56] to compute ground-truth SDF for locations sampled within [-1, 1|3. We then fit a SIREN [66] to the ground-truth SDF generate our initial neural fields. Our main baseline is ARAP [70, 86]. The simplest way to use ARAP to edit neural fields is applying it on a mesh extracted from the imput neural fields using marching cubes [44, 45]. We first present results comparing with ARAP applied to the extracted mesh. Then we will conduct an analysis to show our losses encourage the right behaviors.
Comparing to A ARPP baseline. We follow ARAP [70] to create a set of basic shape deformation operations evaluate our algorithm. In this setting, the user will first specify a set of handles used for manipulation (shown in Figure 5 in blue). In general, the user can apply three basic types of operation on handle: 1) make static (i.e., no deformation), 2) translate it, or rotate it around center. Usually, the first operation is used in combination with the latter two to produce useful deformation. We first show how our algorithm deforms simple objects (e.g., a cylinder and rectangle bar) when the user rotates or translates one end while fixing the other. We then extend this set of operations to shapes with more detail (e.g., Cactus, Armadillo, and Dino). Finally, we test our algorithms when all three operations are specified together on a single shape. The results are shown in Figure 5. We can see that applying ARAP directly on the extracted mesh creates undesirable volume distortion. Our algorithm is able to produce deformation results that are natural while satisfying the user's 's intention.
8

---

(A)
(B)
(C)
(A)
(A)
(B)
(C)
(B)
(C)
(B)
(A)
(B)
(C)
(A)
(B)
(C)
(A)
BB)
Figure 5: Deformation results. (A) Input shape. (B) Baseline. (C) Ours. Red points are user specified handles h, Blue points denote user specified target points t, First row: single rotation or translation on simple shapes. Second and third rows: single rotation or translation more complex objects. Fourth row: multiple edits on complex objects.
Comparing to different ARAP variants. This artifact is partially due to the surface discretization made by the marching cubed algorithm doesn't 't agree with the assumptions made by ARAP [41]. To verify this hypothesis, also include three additional baselines in Figure 6. First, show the results of applying ARAP on the low-poly mesh used to create the input neural fields (Original). Second, we first apply Garland and Heckbert [28] to simplify the surface of the extracted mesh before applying ARAP to it (Remeshed). Finally, we also compare to the results of SR-ARAP [41], which adds smoothness regularization to ARAP remove some unnatural distortion (SR-ARAP). The figure shows that ARAP works much better when the surface discretized appropriately. Discretizing the surface, however, usually requires case-by-case manual adjustment to perform well. Our algorithm does not suffer from such issues as we optimize the thin shell objective directly without committing particular discretization of the surface.
Loss analysis. Here we provide an experiment to analyze the behavior of stretching loss and bending loss. Specifically, we compare models optimized for only the stretching loss L,, only the bending loss Ls. and both losses together. Similar Section 5.2, we deform neural field that approximates the SDF of a 2D rectangle. The results are shown in Figure 7. Optimizing only for L, produces shape that best preserves surface area (or the perimeter the 2D case), which explains the wiggling. Optimizing only for La produces shape that tries to prevent wiggling of the surface, which changes the curvature of the surface. The user can choose appropriate X, and A to produce deformation with resistance to both bending and stretching.

---

User Input
MC
Original
Remeshed
SR-ARAP [41] Ours
K
Figure Additional ARAP baselines. Directly applying ARAP to extracted mesh produces
poor results (i.e., MC). While this can be alleviated by remshing (i.e. Original or Remeshed)
or regularization (i.e. SR-ARAP), this shows mesh deformation algorithms can sensitive to discretization quality. Our method do not discretize the surface, thus circumvent such issue. Red points are handles; Blue points are targets.
7 Discussion, Limitations, and Future work
Advantages of using neural fields for geometry pro- cessing. The main strength of using neural fields for geometry processing that one can disentangle the pro- cessing algorithm from the surface discretization. Users do not need to worry about discretization while manipulat- ing a shape represented in neural fields. This might open the door to more automatic geometry processing pipelines, Additionally, neural fields are modular and can be com- bined with other neural networks. Neural fields can be easily incorporated into current deep learning pipelines. This makes them well-suited for data-driven applications.
C. only
C. + Lb
L. only
Figure 7: bending losses. Blue points are handles; Red points are targets.
Analysis of stretching and
Limitations and future directions. The main limitation of our method is the slow optimization speed that hinders interactive editing. Typically, ARAP can solve each of such deformation problems within half an hour for large meshes, and it can achieve interactive speed for meshes with a small number of triangles. The Taubin smoothing can finish in a couple of seconds. Our deformation method right now requires a TitanX GPU with 12GB memory to train for 10 hours. Our smoothing and sharpening method takes about 10 minutes the same GPU. The current algorithm does not ensure that the output field remains an SDF, which prevents performing multiple editing operations.
We believe many of these limitations can be addressed in future work. One can potentially use techniques like progressive training or meta-learning [67, 75] to improve the training speed. To allow consecutive editing, we can design regularization loss that enforces the output fields to approximate an SDF [4] or design sampling methods robust with noisy SDF. Other interesting directions include providing deeper theoretical analysis and applying neural fields in other geometry processing tasks.
Societal impact. Our work can lead to more efficient geometry processing algorithms. Such algorithms can empower artists generate creative content. Potential negative impact includes misuse of geometry processing algorithms to create offensive content.
8 Conclusion
Our work provides a proof of concept that geometry processing can be done entirely with neural fields without discretizing the surface. We develop network architectures and training objectives for filtering and deforming shapes represented by neural fields and demonstrate the advantages of using neural fields for geometry processing. We hope that our work inspire new generation of geometry processing algorithms using neural fields.
Acknowledgement. Guandao's PhD was supported in part by a research gift from Magic Leap and donation from NVIDIA. We want to thank Wenqi Xian, Professor Steve Marschner, and members of Intel Labs for providing insightful feedback for this project.
10

---

References
[1] M. Alexa, D. Cohen-Or, and Levin. As-rigid-as tergiidas-pposse as-possible shape interpolation. Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 2000. 5,7
[2] Pierre Alliez, Mark Meyer, and Mathieu Desbrun. Interactive geometry remeshing. ACM Transactions on Graphics (TOG), 2133:347-3544 2002.
[3] Pierre Alliez, David Cohen-Steiner, Olivier Devillers, Bruno Lévy, and Mathieu Desbrun. Anisotropic polygonal remeshing. ACM SIGGRAPII 2003 Papers, 2003. 1
[4] Matan Atzmon and Y. Lipman. Sal: Sign agnostic learning of shapes from raw data. 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2562-2571, 2020. 3, 4, 10
[5] Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israclov, Haggai Maron, and Y. Lipman. Controlling neural level sets. In NeurIPS, 2019. 3,5
[6] D. Bechmann. Space deformation models survey, Comput. Graph., 18:571-586, 1994. 2
[7] Jens Behrmann, D. Duvenaud, and J. Jacobsen. Invertible residual networks. In JCML, 2019.
[8] J. Bloomenthal and B. Wyvill. Introduction to implicit surfaces. 1997.
[9] Mario Botsch and O. Sorkine-Hornung. On linear variational surface deformation methods.
IEEE Transactions on Visualization and Computer Graphics, 14:213-230, 2008. 3
[10] Mario Botsch, Mark Pauly, Markus Gross, and Leif P. Kobbelt. Primo: coupled prisms for intuitive surface modeling. In SGP '06, 2006. 2
[1I] Mario Botsch, Leif Kobbelt, Mark Pauly, Pierre Alliez, and Bruno Lévy. Polygon mesh
processing. CRC press, 2010. 1,2
[12] Aljavz Bovzive, Pablo Rodríguez Palafox, Michael Zollhofer, Justus Thies, Angela Dai, and Matthias Nießner. Neural deformation graphs for globally-consistent non-rigid reconstruction. ArXiv, abs/2012.014511, 2020.
[13] Ruojin Cai, Guandao Yang. Hadar Averbuch- Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. 3,
[14] Edwin Catmull. A system for computer generated movies. In Proceedings of the ACM annual conference-Volene 1, pages 422-431, 1972.
[15] Isaac Chao, Ulrich Pinkall, Patrick Sanan, and Peter Schröder. A simple geometric model for elastic deformations. In SIGGRAPH 2010, 2010.
[16] Hunter Chase. Fundamental forms of surfaces and the gauss-bonnet theorem. 2012.
[17] Ricky T. Q. Chen, Jens Rehrmann, D. Duvenaud, and J. Jacobsen. Residual flows for invertible generative modeling, ArXiv, abs/1906.027355. 2019. 6
[18] Xu Chen, Yufeng Zheng, M. Black, Otmar Hilliges, and Andreas Geiger. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. ArXiv, 0ads/104403353. 2021.
3
[19] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939-5948, 2019.
[20] Ming Chuang and M. Kazhdan. Interactive and anisotropic geometry processing using the screened poisson equation. ACM SIGGRAPH 2011 papers, 201 2, 4
[21] Djork-Arné Clevert, Thomas Unterthiner, and Hochreiter. Fast and accurate deep network learning by exponential linear units (clus). arXiv: Learning, 2016. 6
11

---

[22] B. Curless and M. Levoy, A volumetric method for building complex models from range images. Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 1996.
[23] T. Davies, Derek Nowrouzezahrai, and A. Jacobson. On the effectiveness of weight-encoded neural implicit 3d shapes. 2020. 1
[24] A. de Boer, Martijn van der Schoot, and Hester Bijl. Mesh deformation based on radial basis function interpolation. Computers Structures, 85:784-795, 2007.
[25] Yu Deng, Yang, and Xin Tong. Deformed implicit field: Modeling 3d shapes with learned dense correspondence. 2021. 3,
[26] Mathieu Desbrun, M. Meyer, P. Schröder, and A. Barr. Implicit fairing of irregular meshes using diffusion and curvature flow. In SIGGRAPH '99, 1999. 2
[27] Michael Garland and Paul S. Heckbert. Surface simplification using quadric error metrics. Proceedings of the 24th annual conference on Computer graphics and interactive techniques. 1997. 4
[28] Michael Garland and Paul S Heckbert. Surface simplification using quadric enror metries. In Proceedings of the 24th anmual conference on Computer graphics and interactive techniques, pages 209-216, 1997. 9
[29] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7154-7164, 2019. 3
[30] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 4857-4866, 2020.
[31] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Y. Lipman. Implicit geometric regularization for learning shapes. In ICML, 2020. 2, 3,4
[32] Kunal Gupta and Manmohan Chandraker. Neural mesh flow: 3d manifold mesh generationvia diffeomorphic flows. ArXiv, abs/2007.1097. 2020. 3
[33] Zekun Hao, Hadar Averbuch-Elor Noah Snavely, and Serge J. Belongie. Dualsdf: Semantic shape manipulation using a two-level representation. pages 7628-7638, 2020. 3
[34] Jingwei Huang, C. Jiang, Baiqiang Leng, Bin Wang, and L. Guibas. Meshode: A robust and scalable framework for mesh deformation. ArXiv, abs/2005. 1617, 2020.
[35] José A. Iglesias, M. Rumpf, and O. Scherzer. Shape-aware matching of implicit surfaces based on thin shell energies. Foundations of Computational Mathematics (New York, N.y.), 18:891 927, 2018.
[36] A. Jacobson, Baran, Jovan Popovic, and O. Sorkine- Horinne- Bounded biharmonic weights for real-time deformation. Communications of the ACM, 57:99 106, 2014. 2
[37] Timothy Jeruzalski, Boyang Deng, Mohammad Norouzi, J. P. Lewis, G. Hinton, and Andrea Tagliasacchi. Nasa: Neural articulated shape approximation. In ECCV, 2020. 3
[38] Chiyu Max Jiang, Jingwei Huang. Andrea Tagliasacchi, and Leonidas J. Guibas. Shapeflow: Learnable deformation flows among 3d shapes. In NeurIPS, 2020. 3
[39] M. W. Jones, J. A. Bærentzen, and M. Srámek. 3d distance fields: a survey of techniques and applications. IEEE Transactions on Visualization and Computer Graphics, 12:581-599, 2006. 1, 2.
[40] Venkat Krishnamurthy and M. Levoy. Fitting smooth surfaces to dense polygon meshes. Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 1996. 4
12

---

[41] Zohar Levi and C. Gotsman. Smooth rotation enhanced IEEE Transactions on Visualization and Computer Graphics, 21:264 277, 2015.
mesh animation.
10
[42] Y. Lipman, O. Sorkine-Hornung, D. Cohen-Or, D. Levin, Christian RössI, and H. Seidel. Differential coordinates for interactive mesh editing. Proceedings Shape Modeling Applications, 2004., pages 181-190, 2004.
[43] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun- Yan Zhu, and Bryan C. Russell. Editing conditional radiance fields. ArXiv, abs/2105 105.0666,, 2021. 3, 6
[44] W. Lorensen and H. Cline. Marching cubes: A high resolution 3d surface construction algorithm. In SIGGRAPH 87, 1987. 4,
[45] Lars M. Mescheder, Michael Oechsle, M. Niemeyer, S. Nowozin, and Andreas Geiger. Occu- pancy networks: Learning 3d reconstruction function space. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4455-4465, 2019. 1,8
[46] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Deep level sets: Implicit surface representations for 3d shape inference. arXiv preprint arX1v:1901.06802, 2019.
[47] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, J. T. Barron, R. Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020. 1,
[48] Tim Milliron, R. Jensen, Ronen Barzel, and A. Finkelstein. A framework for geometric warps and deformations. ACM Trans. Graph., 21:20-51, 2002. 2
[49] I. Mitchell. The flexible, extensible and efficient toolbox level set methods. Journal of Scientific Computing, 35:300-329, 2008. 2
[50] Henry Moreton and C. Séquin. Functional optimization for fair surface design. Proceedings of the 19th annual conference on Computer graphics and interactive techniques, 1992. 2
[51] K. Museth, D. Breen, Whitaker, and Barr. Level surface editing operators. Proceedings of the 29th annual conference on Computer graphics and interactive techniques, 2002. 1,
[52] Martin E Newell. The utilization of procedure models in digital image synthesis. Technical report, UTAH UNIV SALT LAKE CITY SCHOOL OF COMPUTING, 1975.
[53] M. Niemeyer, Lars M. Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 5378-5388, 2019. 3
[54] S. Osher and Ronald Fedkiw. Level set methods and dynamic implicit surfaces. In Applied mathematical sciences, 2003. 2,
[55] S. Osher and N. Paragios. Geometric level methods in imaging, vision, and graphics. In Springer New York, 2003.
[56] Jeong Joon Park, Peter R. Florence, Straub, Richard A. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 165-174, 2019. 1, 1,4,8 8
[57] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Deformable neural radiance fields. arXiv preprint arXiv:2011.12945 2020. 6
[58] Sida Peng. Junting Dong. Qianqian Wang, Shangzhan Zhang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Animatable neural radiance fields for human body modeling. ArXiv, abs/2105.02872 2021.
[59] H. Piaggio. Differential geometry of curves and surfaces. Nature, 169:560-560, 1952. 3,
13

---

[60] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and F. Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes, ArXiv, abs/201 1. 1.19961, 2020. 3
[61]
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019. 2
[62] Edoardo Remelli, Artem Lukoianov, Stephan R. Richter, Benoit Guillard, Timur M. Bagautdinov, Pierre Baqué, and P. Fua. Meshsdf: Differentiable iso-surface extraction. NeurIPS, 2020.
[63] A. Rosenfeld and Pfaltz. Sequential operations in digital picture processing. J. ACM, 13: 471-494, 1966.
[64] S. Saito, Zeng Huang, R. Natsume, S. Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. 2019 JEEE/CVF International Conference Computer Vision (ICCV), pages 2304 2314, 2019.
[65] T. Sederberg and Parry. Free-form deformation of solid geometric models. Proceedings of the 13th annual conference on Computer graphics and interactive techniques, 1986. 2
[66] V. Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. ArXiv, abs/2006.09661, 2020.
2, 4, 6.
[67] Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. ArXiv, 0ess20006.0966,, 2020. 10
[68] Yang Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. ArXiv, 000119977056000 2019.
[69] Olga Sorkine. Differential representations for mesh processing. In Computer Graphics Forum, volume 25, pages 189-807. Wiley Online Library, 2006. 2
[70] O. Sorkine-Hornung and M. Alexa, As-rigid-as-possibe surface modeling. In Symposium on Geometry Processing, 2007. 2, 7,8
[71] O. Sorkine- Hornung, D. Cohen-Or, Y. Lipman, M. Alexa, Christian Rössl, and H. Seidel. Laplacian surface editing. In SGP '04, 2004. 2
[72] O. Stein, E. Grinspun, M. Wardetzky, and A. Jacobson. Natural boundary conditions for smoothing in geometry processing. ACM Transactions on Graphics (TOG), 37:1 - 13, 2018.
[73] Towaki Takikawa, Joey Litalien, Yin, K. Kreis, Charles Loop, Derek Nowrouzezahrai, A. Jacobson, M. McGuire, and S. Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. ArXiv, abs/2101. 10994, 2021.
[74] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, R. Ramamoorthi, J. T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. ArXiv, abs/2006. 107,9900..17,,, 2020. 2, 6
[75] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate- based neural representa- tions. In CVPR, 2021. 10
[76] G. Taubin. A signal processing approach to fair surface design. Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, 1995. 2, 4
[77] Demetri Terzopoulos, John C. Platt, A. Barr, and K. Fleischer. Elastically deformable models. In SIGGRAPH '87, 1987.
[78] Jason Torchinsky.
The first real object ever 3D scanned and ren- dered was a VW beetle. Jalopnik, 2013. https://jaalonni com/ che-first--o rew-st--re---oeet-e-eee-e-- -dver--d- -nd-r-eeeededd---------e-e---
14

---

[79] B. Vallet and B. Lévy. Spectral geometry processing with manifold harmonics. Computer Graphics Forum, 27, 2008. 2
[80] Weiyue Wang, D. Ceylan, R. Mech, and U. Neumann. 3dm: 3d deformation network. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1038-1046. 2019.
[81] Yifan Wang, Noam Aigerman, Vladimir G. Kim, S. Chaudhuri, and O. Sorkine-Hornuny 2020 IEEE/CVF Conference Computer
Neural cages for detail- preaevipresr 3d deformations. Vision and Pattern Recognition (CVPR), pages 72-80, 2020. 3
[82] William Welch and A. Witkin, Variational surface modeling. Proceedings of the 19th annual conference on Computer graphics and interactive techniques, 1992.
[83] Zhan Xu, Yang Zhou, E. Kalogerakis, Chris Landreth, and Karan Singh, Rignet: Neural rigging for articulated characters. ArXiv, abs/2005.00559, 2020. 3
[84] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33, 2020.
[85] Zerong Zheng, Tao Yu, Qionghai Dai, and Y. Liu. Deep implicit templates for 3d shape representation. ArXiv, abs/201 1. 14565, 2020. 3
[86] Qian- Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv: 098477 2018.
15


Reviews:
[{"review_id": "zZTvgZ8MQa8", "reviewer": "Reviewer_7RWH", "summary": "The paper explores the use of neural implicit fields for classical geometry processing operations. In doing so, it addresses the two fundamental issues that make this non-trivial (1) the identification of losses that do not require local parametrization domains over which integration of differential properties is usually performed, and (2) the sampling of neural implicit fields on the surface for the evaluation of such losses.", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI am extremely excited about this paper, and had hoped to work on this myself if I had enough human resources available. However, it turns out it was actually more difficult than I expected (see comments in my summary), and the authors did an excellent job at addressing the technical challenges it presented. So overall, I am extremely happy about the idea pitch, overall convinced by the method section (this is because I must admit I do not have the time to triple check the derivations in the supplementary! which is also why I mark my review with lower confidence), and mostly satisfied with the evaluations (see below). Overall, I would expect this to be a capstone paper that will attract thousands of citations in the future, despite all the limitations the authors correctly point out, and I am excited to see it accepted.\n\nI only have a few remarks/questions for the authors, which I hope can be addressed, but that I do not think would affect my rating (the key factor in championing/rejecting the paper is the correctness of the derivations, to which I ought to delegate to another reviewer... or the AC).\n\n\n## core questions\n1. In my first read, I was taken by surprise by technical \"jumps\", only to realize the derivations were in the supplementary... it would be an excellent idea to explicitly refer to sub-sections in the supplementary in the main paper (and I hope the two will be concatenated in a likely future arXiv submission!)\n1. You compare ARAP-like deformations on meshes for marching cubes. However, MC meshes have elements with terrible finite-element conditioning, and nobody in his right mind would execute polygonal mesh processing algorithms on such input. I think it would be a good idea to include comparisons to isotropically re-meshes version of MC meshes (Botsch et al.) alongside the \"raw\" MC comparison you currently have?\n1. L91: I think you are missing a relevant work in NIFs deformation, especially as it's related to the invertibility you employ in your current solution → \"ShapeFlow: Learnable Deformations Among 3D Shapes\" @ NeurIPS 2020\n1. L150-L153: I am not quite convinced about this argument, which is made even worse by the fact that there is no ablation/experiment to verify its effectiveness. Are you doing this because, since you are executing some neural form of \"explicit Euler\" mean-curvature flow, this results in instability in the smoothing operation during optimization? This is particularly important as \"sharp\" meshes processed \"Implicit Fairing\" work with no problems, which reveals a flaw/weakness in your algorithm?\n1. Sec5.3: what's \"mixing time\"? also, with a perfect SDF (9) would converge in a single iteration, so why would that be a problem? Further, I am a bit confused about this section. My understanding is that SDF-projections of ambient samples would overall results in an over-sampling of large positive curvature areas, and an under-sampling of large negative curvature areas (which Fig.2 seems to confirm). However, I do not recall this being discussed in the text, and the \"rejection\" sampling you provide is a simple (yet somehow effective) \"hack\" to resolve this problem. Further, I think you are missing comparisons showing that Langevin on the *deformed* NIF does not work (both with and without rejection sampling?). \n1. L275: I think you must justify why it is necessary to employ this \"constraint satisfaction\" threshold, I am not fully convinced why it would be necessary. Also, why did you not use the penalty method instead of AG, did the simpler penalty method just fail, or was it too hard to tune?\n1. Fig.5: I am not quite sure what L_s+L_b is... isn't that the full model? And if so, why does it perform so poorly? Because you picked wrong loss-balance hyperparameters? (I was expecting something better than L_b)\n\n## minor points\n1. L120: I was confused by this statement, as $P$ does not measure tangential stretch, it's a tool that combined with other quantities (later in the paper) allows you to understand tangential stretch\n1. L135(equation number missing): I would strongly advise not to use $k$ and $\\kappa$, as my eyes crossed while reading your math. Could you use $\\lambda$ or $\\omega$ as a symbol to balance different losses?\n1. Would be a lot better if the results for the smoothing section were presented right away... especially as it's just one figure and ~10 lines of text?\n1. I would much rather read the limitations in the main paper than in the supplementary material -- it is one of the most important sections of any paper! It would be more tasteful to move the background recap on differential geometry to supplementary instead.\n1. What does the \"r\" in CDr and EMDr mean?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations stated, but should be moved to the main paper.\n\nI had a good chuckle when I read the societal impact paragraph (thank you authors!). I think it's important for NeurIPS to have this sections for some papers (e.g. generative face/text models, face replacement, etc...) but asking it from every papers (e.g. this one) is just plain silly. This is, of course, a commentary towards NeurIPS, and not towards the authors.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Wb8R9LKKLQn", "reviewer": "Reviewer_29Uu", "summary": "This paper proposes a method to perform some common operations from geometry processing on shapes represented as neural implicit fields (i.e., deep SDFs) rather than parameterized meshes. In particulars, by utilizing the fact that high order derivatives of NIFs can be computed automatically, the authors formulate several loss terms, which allow them to apply sharpening, smoothing, or deformation to an existing implicit field via a reoptimization. They show qualitative and qualitative results and comparisons to the mesh-based analogs of their algorithm and present a fairly comprehensive ablation study validating their design choices.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "With deep implicit 3D shape representations becoming more and more expressive and popular, it is important to develop algorithms for making them useful for artists and designers who desire direct control over geometry. The proposed method is a nice first step towards developing geometry processing tools that operate on NIFs rather than meshes. I think the paper is very well written and clearly organized, shows nice experimental results, and motivates a lot of interesting and useful directions for future work. Below are some suggestions and questions about the method and exposition.\n\nI don't entirely understand why the smoothing/sharpening losses are evaluated at points below a curvature threshold (as described in Section 4). If SIREN isosurfaces exhibit so much high-frequency noise, which points remain after the filtering? It would be good to include some clarification about this and possibly a figure illustrating which points get filtered out.\n\nSome of the equations in Section 5.2 are very densely packed in the text and difficult to read---especially, e.g., the inline 2x2 matrix on line 231. It would be okay to move some of these details to the Supplementary Material, but spacing should be added for readability.\n\nWhile evaluating baselines on meshes extracted from the implicit fields makes some sense, it would also make sense to show the mesh-based methods applied to the original input meshes. Currently, the process of mesh -> implicit field -> mesh is a bit roundabout and not entirely fair, since the proposed method is only evaluated on implicit fields that originate from meshes.\n\nIs it possible to incorporate these geometric operations on NIFs into a data-driven learning pipeline in order to avoid the expensive optimization for each edit? For instance, can some of the same loss functions be used to learn these operations generically rather than on a specific input?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Both are adequately discussed in Section 7 as well as in the Supplementary.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "V8getCjD_QT", "reviewer": "Reviewer_1jPU", "summary": "This submission introduces methods to smooth, sharpen and deform shapes represented as neural implicit fields (NIF) without meshing them first.\nSmoothing and sharpening are done by training a new NIF with the same SDF values and the proposed regularization loss to increase or decrease curvature.\nDeformations are obtained by applying an invertible deformation field that warps 3D space to match deformation constraints. In this regard, the contributions of the paper are:\n- Representing the deformation field as an invertible network, with positional encoding\n- Introducing 2 regularization terms to preserve shape properties: stretching and bending losses, that respectively penalize change of dot product and curvature on the surface.\n- Introducing a uniform sampling method on the shape surface to apply these stretching and bending losses.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Originality:__ the presented toolbox is novel and allows to manipulate 3D shapes from their implicit representations. Related works on neural implicit representations of 3D shape are reported, and none of them allows to manipulate shapes without having to mesh them first. In this perspective, the proposed method is novel.\n\n\n__Significance:__ the *overall aim of the paper not entirely clear* to me: what is the advantage of processing shapes in their implicit representations? The abstract sells the idea that “Meshes are hard to optimize for topology change“, and optimizing a NIF should indeed allow for topology change. But this is neither demonstrated in the paper, nor re-emphasized later. So the **abstract is a bit misleading.**\n\nThe introduction then advertises the use of “higher order derivatives” for deforming shapes, but again this is not clear how or when.\nOverall, as stated at the end of Section 2, the presented “method avoids iso-surface extraction”, but what is the benefit of doing so? It also seems that iso-surface extraction is anyways required to get the shape in an explicit form, be it before or after processing.\nThe real value of the presented method to me is the fact it decorrelates meshing resolution from shape manipulation parameters, as illustrated from results in 6.1. More emphasis should be placed on this.\n\nSimilarly, on line 163, the difference between the proposed method compared to [22, 39, 53] could be stressed for a more impactful submission.\n\nTo increase the significance of this work, a demonstration of how it can be applied to different NIF architectures (not only SIRENS) and in the case of a learned shape prior (ex. DeepSDF, ie. apply regularization at training time) would be welcome.\n\nAnother concern is the evaluation against baseline method ARAP is not consistent with the originally reported figures in [65]. In Fig. 7 of [65] the cactus is not undergoing the undesirable volumetric deformation presented here and bends smoothly. How can this be explained?\n\n__Quality:__ The ablation study successfully demonstrates the effectiveness of all the proposed components, and visual results are convincing. It is difficult to provide metrics to evaluate deformation and smoothing quality. However,  Tab. 1 from the supplementary showing volume vs surface preservation is interesting and could probably moved to the main paper.\n\n__Clarity:__ figures and mathematical formalism are neat and well written for easily understanding the proposed method, with the exception of 5.2.1 and 5.2.2. They both introduce a formalism based on an available x(u,v) parametrization to compute matrix B. But since x(u,v) is not available, this formalism is then discarded for another one, which is compatible with NIF representations. But why introduce B and the associated formalism in the first place? It does not seem to help building and intuition, and could even be confusing.\n\nAnother point that is not very clear is on line 290 about baselines: why rerun the data preparation pipeline to try the processed mesh back into an NIF? What is the goal of getting an NIF back as output?\n\nOverall, this submission is very complete technically speaking and just enough details are given such that a careful reader could probably re-implement the method.\n\n__Cosmetic remarks:__\n- line 115 n_d(x) is used to refer to the “surface normal” of a point that is not on the surface… Shouldn’t n_d(.) rather be defined as the SDF field gradient? So that it has a sense to manipulate it for points not on the surface.\n- Line 150 : “curvature regularization” would sound better than “curvature mismatch”\n- Line 211: x_u, x_v have not been introduced.\n- Line 263: “is achieveD by […] keepING”\nIn the supplementary:\n- Parentheses are messed up in the intro\n- Line 87 “inteRgration”\n- Line 131 “turths”\n \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The submission is honest about a main limitation of the proposed method: it is slow and does not allow for interactive editing. The supplementary material reports a 5 to 10 hours optimization time for deforming a mesh. This should maybe be reported in the main paper, along with the time required by the baseline ARAP.\nOverall, it would be valuable to present the entire Discussion section of the supplementary material in the main paper.\n\nSocietal impact is correctly assessed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5 hours"}, "extra_text": {}}, {"review_id": "PSnEwb1jcq", "reviewer": "Reviewer_8TN2", "summary": "This work introduces a framework for geometry processing on surfaces represented as neural networks (neural implicit fields). \nNamely, authors design two algorithms: one for surface smoothing, and one for surface deformation with control points (deformation handles). The core idea behind both is to introduce a new neural network (for smoothing another neural implicit field and for deformation an \"invertible deformation field\"), and optimize it with gradient descent, using a set of losses defined directly on implicit surface. Qualitative experimental evaluation is conducted on a set of meshes, for deformation comparison is provided to a single baseline: as-rigid-as-possible deformation, and for smoothing to Taubin smoothing. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Writing quality / clarity\n\n**+** The paper is really well written: there is enough background provided to make the manuscript self-contained, and proposed methods are discussed in sufficient detail and explained well. \n\n**+/-** Authors also do a good job at referring to existing work on neural implicit representations, but omit some of important references for mesh deformations (e.g. [1]-[4]). \n\n### Contributions / significance\n\n**+** On the one hand, this line of work is certainly worth investigating: neural implicit representations have been used successfully for geometry modeling as well as for some other relevant applications (e.g. neural rendering), so having reliable methods for surface smoothing and deformation could be quite useful. \n\n**-** On the other hand, it is doubtful that the methods presented in this work are at all practical: they rely on heavy optimization, and effectively require training an additional neural network for each deformation / smoothing. This, as authors themselves admit in their limitations paragraph, probably makes interactive use of proposed method highly impractical. Considering that most of the applications would either be real-time / interactive or use mesh smoothing / deformation as a basic building block (e.g. when sampling novel shapes for shape optimiziation), it is unlikely that the framework presented in this work can actually be used in practice. Moreover, in some applications (again, e.g. shape optimization), it is often very useful for the underlying parameterization or deformation algorithm to be differenitable, which seems to be hard to achieve when the deformation itself is a heavy optimization process.\n\n**-** Neural implicit representations can be used to represent a _collection_ of shapes ([22], [25]), in which case conditioning signals or latent space can be used as an optimizable parameter space and similar constraints can be used as an objective. I do not find the argument (L97) \"it’s unclear how to apply these methods to out-of-distribution shapes\" very convincing, because proposed method in a way is even less generalizable, as it is only operating on a single sample. \n\n**+** Formulation of the stretching and bending losses for implicit surfaces from Section 5.2 will probably be interesting to the community.\n\n\n### Evaluation \n\nThere are multiple concerns that I have about the evaluation.\n\n**-** There are several alternatives to ARAP (see references below for non-exhaustive list, there is also RBF), some of which more recent, which are not reported by the authors. Moreover, results of ARAP seem to be very different from the original manuscript (e.g. on the same \"cactus\" shape). More generally, it is a bit hard to judge the quality of the algorithm from such a limited number of static examples and few baseline comparisons. \n\n**-** It would be interesting to see how methods that directly operate on implicit surfaces work on these tasks (some of those referred to on L88-L92).\n\n**-** The weakest point of the proposed framework is how expensive it is. Yet, there is no comparison in terms of speed or memory requirements with respect to the baselines.\n\n**-** (minor) I am not very convinced that it is necessary to compare to the surfaces automatically extracted from the NIF, rather than original meshes: especially since for most of the reported examples they are available. \n\n**-** (minor) It was a bit strange that authors claimed the benefits of implicit representations as being able to optimize for topology changes, and then report results on topology-preserving deformations. \n\n\n### References\n[1] Botsch et al '2006: \"Primo: coupled prisms for intuitive surface modeling”\n\n[2] Chao et al '2010: “A simple geometric model for elastic deformations”\n\n[3] Crane et al '2011: \"Spin Transformations of Discrete Surfaces\"\n\n[4] Levi et al '2014: \"Smooth rotation enhanced as-rigid-as-possible mesh animation\"\n\n[5] de Boer et al '2007: \"Mesh deformation based on radial basis function interpolation\"\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors shortly discuss limitations in their work, including the slowness due to the fact that the method relies on optimization, which indeed can cause difficulties for interactive editing as well as applicability of the method as a backbone for downstream tasks.  However, since the practical implications of those can be very severe, a more extensive formal evaluation and more context in the form of comparisons and timings (on similar hardware) would be necessary for readers to fully understand the scope of these issues.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper produced a substantial amount of discussion before and after the rebuttal, but in the end the decision was to accept this work.  Some readers (notably reviewer 7RWH) were excited about the vision presented in this work and possibilities for extension, while some others had concerns about efficiency/practicality.

In the camera-ready, please more explicit in discussing the limitations of the work (e.g. highlighting that it takes 5 hours for a task that typically can be done in real-time), as well as provide a fair comparison to the baseline to avoid misleading the readers (e.g. running ARAP on essentially degraded meshes).
Decision: Accept (Poster)