Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method to perform some common operations from geometry processing on shapes represented as neural implicit fields (i.e., deep SDFs) rather than parameterized meshes. In particulars, by utilizing the fact that high order derivatives of NIFs can be computed automatically, the authors formulate several loss terms, which allow them to apply sharpening, smoothing, or deformation to an existing implicit field via a reoptimization. They show qualitative and qualitative results and comparisons to the mesh-based analogs of their algorithm and present a fairly comprehensive ablation study validating their design choices.
main_review: With deep implicit 3D shape representations becoming more and more expressive and popular, it is important to develop algorithms for making them useful for artists and designers who desire direct control over geometry. The proposed method is a nice first step towards developing geometry processing tools that operate on NIFs rather than meshes. I think the paper is very well written and clearly organized, shows nice experimental results, and motivates a lot of interesting and useful directions for future work. Below are some suggestions and questions about the method and exposition.

I don't entirely understand why the smoothing/sharpening losses are evaluated at points below a curvature threshold (as described in Section 4). If SIREN isosurfaces exhibit so much high-frequency noise, which points remain after the filtering? It would be good to include some clarification about this and possibly a figure illustrating which points get filtered out.

Some of the equations in Section 5.2 are very densely packed in the text and difficult to read---especially, e.g., the inline 2x2 matrix on line 231. It would be okay to move some of these details to the Supplementary Material, but spacing should be added for readability.

While evaluating baselines on meshes extracted from the implicit fields makes some sense, it would also make sense to show the mesh-based methods applied to the original input meshes. Currently, the process of mesh -> implicit field -> mesh is a bit roundabout and not entirely fair, since the proposed method is only evaluated on implicit fields that originate from meshes.

Is it possible to incorporate these geometric operations on NIFs into a data-driven learning pipeline in order to avoid the expensive optimization for each edit? For instance, can some of the same loss functions be used to learn these operations generically rather than on a specific input?
limitations_and_societal_impact: Both are adequately discussed in Section 7 as well as in the Supplementary.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work introduces a framework for geometry processing on surfaces represented as neural networks (neural implicit fields). 
Namely, authors design two algorithms: one for surface smoothing, and one for surface deformation with control points (deformation handles). The core idea behind both is to introduce a new neural network (for smoothing another neural implicit field and for deformation an "invertible deformation field"), and optimize it with gradient descent, using a set of losses defined directly on implicit surface. Qualitative experimental evaluation is conducted on a set of meshes, for deformation comparison is provided to a single baseline: as-rigid-as-possible deformation, and for smoothing to Taubin smoothing. 
main_review: ### Writing quality / clarity

**+** The paper is really well written: there is enough background provided to make the manuscript self-contained, and proposed methods are discussed in sufficient detail and explained well. 

**+/-** Authors also do a good job at referring to existing work on neural implicit representations, but omit some of important references for mesh deformations (e.g. [1]-[4]). 

### Contributions / significance

**+** On the one hand, this line of work is certainly worth investigating: neural implicit representations have been used successfully for geometry modeling as well as for some other relevant applications (e.g. neural rendering), so having reliable methods for surface smoothing and deformation could be quite useful. 

**-** On the other hand, it is doubtful that the methods presented in this work are at all practical: they rely on heavy optimization, and effectively require training an additional neural network for each deformation / smoothing. This, as authors themselves admit in their limitations paragraph, probably makes interactive use of proposed method highly impractical. Considering that most of the applications would either be real-time / interactive or use mesh smoothing / deformation as a basic building block (e.g. when sampling novel shapes for shape optimiziation), it is unlikely that the framework presented in this work can actually be used in practice. Moreover, in some applications (again, e.g. shape optimization), it is often very useful for the underlying parameterization or deformation algorithm to be differenitable, which seems to be hard to achieve when the deformation itself is a heavy optimization process.

**-** Neural implicit representations can be used to represent a _collection_ of shapes ([22], [25]), in which case conditioning signals or latent space can be used as an optimizable parameter space and similar constraints can be used as an objective. I do not find the argument (L97) "it’s unclear how to apply these methods to out-of-distribution shapes" very convincing, because proposed method in a way is even less generalizable, as it is only operating on a single sample. 

**+** Formulation of the stretching and bending losses for implicit surfaces from Section 5.2 will probably be interesting to the community.


### Evaluation 

There are multiple concerns that I have about the evaluation.

**-** There are several alternatives to ARAP (see references below for non-exhaustive list, there is also RBF), some of which more recent, which are not reported by the authors. Moreover, results of ARAP seem to be very different from the original manuscript (e.g. on the same "cactus" shape). More generally, it is a bit hard to judge the quality of the algorithm from such a limited number of static examples and few baseline comparisons. 

**-** It would be interesting to see how methods that directly operate on implicit surfaces work on these tasks (some of those referred to on L88-L92).

**-** The weakest point of the proposed framework is how expensive it is. Yet, there is no comparison in terms of speed or memory requirements with respect to the baselines.

**-** (minor) I am not very convinced that it is necessary to compare to the surfaces automatically extracted from the NIF, rather than original meshes: especially since for most of the reported examples they are available. 

**-** (minor) It was a bit strange that authors claimed the benefits of implicit representations as being able to optimize for topology changes, and then report results on topology-preserving deformations. 


### References
[1] Botsch et al '2006: "Primo: coupled prisms for intuitive surface modeling”

[2] Chao et al '2010: “A simple geometric model for elastic deformations”

[3] Crane et al '2011: "Spin Transformations of Discrete Surfaces"

[4] Levi et al '2014: "Smooth rotation enhanced as-rigid-as-possible mesh animation"

[5] de Boer et al '2007: "Mesh deformation based on radial basis function interpolation"


limitations_and_societal_impact: Authors shortly discuss limitations in their work, including the slowness due to the fact that the method relies on optimization, which indeed can cause difficulties for interactive editing as well as applicability of the method as a backbone for downstream tasks.  However, since the practical implications of those can be very severe, a more extensive formal evaluation and more context in the form of comparisons and timings (on similar hardware) would be necessary for readers to fully understand the scope of these issues.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This submission introduces methods to smooth, sharpen and deform shapes represented as neural implicit fields (NIF) without meshing them first.
Smoothing and sharpening are done by training a new NIF with the same SDF values and the proposed regularization loss to increase or decrease curvature.
Deformations are obtained by applying an invertible deformation field that warps 3D space to match deformation constraints. In this regard, the contributions of the paper are:
- Representing the deformation field as an invertible network, with positional encoding
- Introducing 2 regularization terms to preserve shape properties: stretching and bending losses, that respectively penalize change of dot product and curvature on the surface.
- Introducing a uniform sampling method on the shape surface to apply these stretching and bending losses.
main_review: __Originality:__ the presented toolbox is novel and allows to manipulate 3D shapes from their implicit representations. Related works on neural implicit representations of 3D shape are reported, and none of them allows to manipulate shapes without having to mesh them first. In this perspective, the proposed method is novel.


__Significance:__ the *overall aim of the paper not entirely clear* to me: what is the advantage of processing shapes in their implicit representations? The abstract sells the idea that “Meshes are hard to optimize for topology change“, and optimizing a NIF should indeed allow for topology change. But this is neither demonstrated in the paper, nor re-emphasized later. So the **abstract is a bit misleading.**

The introduction then advertises the use of “higher order derivatives” for deforming shapes, but again this is not clear how or when.
Overall, as stated at the end of Section 2, the presented “method avoids iso-surface extraction”, but what is the benefit of doing so? It also seems that iso-surface extraction is anyways required to get the shape in an explicit form, be it before or after processing.
The real value of the presented method to me is the fact it decorrelates meshing resolution from shape manipulation parameters, as illustrated from results in 6.1. More emphasis should be placed on this.

Similarly, on line 163, the difference between the proposed method compared to [22, 39, 53] could be stressed for a more impactful submission.

To increase the significance of this work, a demonstration of how it can be applied to different NIF architectures (not only SIRENS) and in the case of a learned shape prior (ex. DeepSDF, ie. apply regularization at training time) would be welcome.

Another concern is the evaluation against baseline method ARAP is not consistent with the originally reported figures in [65]. In Fig. 7 of [65] the cactus is not undergoing the undesirable volumetric deformation presented here and bends smoothly. How can this be explained?

__Quality:__ The ablation study successfully demonstrates the effectiveness of all the proposed components, and visual results are convincing. It is difficult to provide metrics to evaluate deformation and smoothing quality. However,  Tab. 1 from the supplementary showing volume vs surface preservation is interesting and could probably moved to the main paper.

__Clarity:__ figures and mathematical formalism are neat and well written for easily understanding the proposed method, with the exception of 5.2.1 and 5.2.2. They both introduce a formalism based on an available x(u,v) parametrization to compute matrix B. But since x(u,v) is not available, this formalism is then discarded for another one, which is compatible with NIF representations. But why introduce B and the associated formalism in the first place? It does not seem to help building and intuition, and could even be confusing.

Another point that is not very clear is on line 290 about baselines: why rerun the data preparation pipeline to try the processed mesh back into an NIF? What is the goal of getting an NIF back as output?

Overall, this submission is very complete technically speaking and just enough details are given such that a careful reader could probably re-implement the method.

__Cosmetic remarks:__
- line 115 n_d(x) is used to refer to the “surface normal” of a point that is not on the surface… Shouldn’t n_d(.) rather be defined as the SDF field gradient? So that it has a sense to manipulate it for points not on the surface.
- Line 150 : “curvature regularization” would sound better than “curvature mismatch”
- Line 211: x_u, x_v have not been introduced.
- Line 263: “is achieveD by […] keepING”
In the supplementary:
- Parentheses are messed up in the intro
- Line 87 “inteRgration”
- Line 131 “turths”
 

limitations_and_societal_impact: The submission is honest about a main limitation of the proposed method: it is slow and does not allow for interactive editing. The supplementary material reports a 5 to 10 hours optimization time for deforming a mesh. This should maybe be reported in the main paper, along with the time required by the baseline ARAP.
Overall, it would be valuable to present the entire Discussion section of the supplementary material in the main paper.

Societal impact is correctly assessed.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper explores the use of neural implicit fields for classical geometry processing operations. In doing so, it addresses the two fundamental issues that make this non-trivial (1) the identification of losses that do not require local parametrization domains over which integration of differential properties is usually performed, and (2) the sampling of neural implicit fields on the surface for the evaluation of such losses.
main_review: 
I am extremely excited about this paper, and had hoped to work on this myself if I had enough human resources available. However, it turns out it was actually more difficult than I expected (see comments in my summary), and the authors did an excellent job at addressing the technical challenges it presented. So overall, I am extremely happy about the idea pitch, overall convinced by the method section (this is because I must admit I do not have the time to triple check the derivations in the supplementary! which is also why I mark my review with lower confidence), and mostly satisfied with the evaluations (see below). Overall, I would expect this to be a capstone paper that will attract thousands of citations in the future, despite all the limitations the authors correctly point out, and I am excited to see it accepted.

I only have a few remarks/questions for the authors, which I hope can be addressed, but that I do not think would affect my rating (the key factor in championing/rejecting the paper is the correctness of the derivations, to which I ought to delegate to another reviewer... or the AC).


## core questions
1. In my first read, I was taken by surprise by technical "jumps", only to realize the derivations were in the supplementary... it would be an excellent idea to explicitly refer to sub-sections in the supplementary in the main paper (and I hope the two will be concatenated in a likely future arXiv submission!)
1. You compare ARAP-like deformations on meshes for marching cubes. However, MC meshes have elements with terrible finite-element conditioning, and nobody in his right mind would execute polygonal mesh processing algorithms on such input. I think it would be a good idea to include comparisons to isotropically re-meshes version of MC meshes (Botsch et al.) alongside the "raw" MC comparison you currently have?
1. L91: I think you are missing a relevant work in NIFs deformation, especially as it's related to the invertibility you employ in your current solution → "ShapeFlow: Learnable Deformations Among 3D Shapes" @ NeurIPS 2020
1. L150-L153: I am not quite convinced about this argument, which is made even worse by the fact that there is no ablation/experiment to verify its effectiveness. Are you doing this because, since you are executing some neural form of "explicit Euler" mean-curvature flow, this results in instability in the smoothing operation during optimization? This is particularly important as "sharp" meshes processed "Implicit Fairing" work with no problems, which reveals a flaw/weakness in your algorithm?
1. Sec5.3: what's "mixing time"? also, with a perfect SDF (9) would converge in a single iteration, so why would that be a problem? Further, I am a bit confused about this section. My understanding is that SDF-projections of ambient samples would overall results in an over-sampling of large positive curvature areas, and an under-sampling of large negative curvature areas (which Fig.2 seems to confirm). However, I do not recall this being discussed in the text, and the "rejection" sampling you provide is a simple (yet somehow effective) "hack" to resolve this problem. Further, I think you are missing comparisons showing that Langevin on the *deformed* NIF does not work (both with and without rejection sampling?). 
1. L275: I think you must justify why it is necessary to employ this "constraint satisfaction" threshold, I am not fully convinced why it would be necessary. Also, why did you not use the penalty method instead of AG, did the simpler penalty method just fail, or was it too hard to tune?
1. Fig.5: I am not quite sure what L_s+L_b is... isn't that the full model? And if so, why does it perform so poorly? Because you picked wrong loss-balance hyperparameters? (I was expecting something better than L_b)

## minor points
1. L120: I was confused by this statement, as $P$ does not measure tangential stretch, it's a tool that combined with other quantities (later in the paper) allows you to understand tangential stretch
1. L135(equation number missing): I would strongly advise not to use $k$ and $\kappa$, as my eyes crossed while reading your math. Could you use $\lambda$ or $\omega$ as a symbol to balance different losses?
1. Would be a lot better if the results for the smoothing section were presented right away... especially as it's just one figure and ~10 lines of text?
1. I would much rather read the limitations in the main paper than in the supplementary material -- it is one of the most important sections of any paper! It would be more tasteful to move the background recap on differential geometry to supplementary instead.
1. What does the "r" in CDr and EMDr mean?
limitations_and_societal_impact: Limitations stated, but should be moved to the main paper.

I had a good chuckle when I read the societal impact paragraph (thank you authors!). I think it's important for NeurIPS to have this sections for some papers (e.g. generative face/text models, face replacement, etc...) but asking it from every papers (e.g. this one) is just plain silly. This is, of course, a commentary towards NeurIPS, and not towards the authors.
review_title: null

Meta-review: This paper produced a substantial amount of discussion before and after the rebuttal, but in the end the decision was to accept this work.  Some readers (notably reviewer 7RWH) were excited about the vision presented in this work and possibilities for extension, while some others had concerns about efficiency/practicality.

In the camera-ready, please more explicit in discussing the limitations of the work (e.g. highlighting that it takes 5 hours for a task that typically can be done in real-time), as well as provide a fair comparison to the baseline to avoid misleading the readers (e.g. running ARAP on essentially degraded meshes).
Decision: Accept (Poster)