Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a method for interactively editing 3D shapes defined as the level sets of neural implicit functions. Specifically, the approach quantifies the distribution of changes to this level set (the shape boundary) w.r.t. the parameters governing the neural implicits (either the network weights for a model regressed to fit a single shape, or the latent variables of a generative model). This is referred to as "boundary sensitivity". Based on this, the authors demonstrate several example of shape editing guided by high-level sparse user input, as well as introducing deformation constraints (rigidity, volume preservation...) into the framework. While this sort of inverse control has been demonstrated for other representations, this is, to the best of my knowledge, a novel contribution for neural implicits.

strength_and_weaknesses: The paper is well-written and develops an interesting method for computing the boundary sensitivity of an implicit field. Various applications and examples are presented to highlight the utility of the approach. In general, I am positive about this paper.

One weakness is that the paper does not really go into how and where the method fails. E.g. for what magnitude of displacements does the linearity assumption lead to incorrect results? If the network is very complex (very high-dimensional), are the geometric edits induced by perturbing network weights still smooth enough to look plausible? How does this latter aspect play with the explicit constraints induced in Sections 4.3 and 4.4?

Another weakness is that it does not sufficiently compare to baselines, as pointed out in other reviews.
clarity,_quality,_novelty_and_reproducibility: The paper is fairly clear. The work appears to be high-quality and original.

I was curious if large user-specified local edits can be handled by dividing them into small incremental changes and then time-stepping the method, i.e. to move the bunny's ear by 10 units, break it into 10 steps of 1 unit each and recursively apply the method to obtain the final integrated deformation of the bunny.

summary_of_the_review: I think the paper develops a nice mathematically justified approach for tying implicit field boundary changes to parameter updates, using this to solve various inverse control tasks. I think it can be better positioned w.r.t. evaluations and prior work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a unified story for shape derivatives of neural implicit surfaces. The method enables projection onto deformations that preserve volume or are approximately rigid, and it suggests a path toward enforcing other kinds of constraints. Results are limited and only qualitative.
strength_and_weaknesses: ## Strengths
- The paper is mostly clear and concise
## Weaknesses
- The paper has no quantitative empirical results. As such, it is unclear how well, e.g., volume preservation works. Since the method is only preserving volume to first order by using step-projections, this should be evaluated.
- Ditto for the AKVF approach to rigidity
clarity,_quality,_novelty_and_reproducibility: ## Novelty
- The paper is based on classical sensitivity analysis for implicit surfaces, which is not new.
- The application to neural fields is also not new (Mehta et al. 2022), though its use without any mesh may be new.

## Clarity
- The manner in which user input is provided is unclear. In section 3, "Let $\delta \overline{\mathbf{x}}$ ... be a prescribed deformation on (a part of) the boundary" suggests that the user input is provided only at points on the surface. However, points on the surface are sampled with rejection sampling. So presumably the user input must be provided on a region of positive volume, in the neighborhood of part of the surface. This needs to be more carefully explained.
- The iterative scheme presented in section 4 needs to be explained better. Is new user input provided at each iteration? Or is the iteration meant to converge to the user's target displacement? How many iterations are needed?
- Section 4.4 is really about the method and should be moved to the end of section 3 and integrated with section 3. The second paragraph of section 4, describing the iterative displacement scheme, should also be moved into section 3 and expanded (see above).
- In the "semantic editing" application presented in Figure 4, it is not clear from the discussion whether the method optimizes the latent codes and weights together, or only the weights.
- In equation (8), it is unclear how $\int_\Omega \mathrm{div}(h\delta\mathbf{x}) \mathrm{d}x$ becomes $\delta \mathbf{\Theta}^\top \int_\Omega \mathbf{b} \mathrm{d}x$.
- The paper claims "the upper limit to how fine an arbitrary detail anywhere on the shape can be is given by $A/P$ where $A$ is the surface area of the shape" (p. 5). This certainly seems intuitively reasonable, but to claim it as certain more justification should be presented.

## Minor points
- In Figure 3: "show segmentation" is unclear
- Page 6: remove "Since this is valid for interpolation to many samples,"
- Page 6: "each basis" => "each basis function" (2x)
- Page 6: "allows to intuitively traverse" => "intuitively traverses"
- $\mathbb{R}^{3,3}$ => $\mathbb{R}^{3\times 3}$ and similarly for $\mathbb{R}^{3,3,P}$
- Page 7: "several choices to recovering" => "several choices for recovering"


summary_of_the_review: The paper's fully-implicit approach to editing neural fields seems valuable. More careful evaluation and explanation is required to make this a great paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The key idea behind this work is to leverage boundary sensitivity to express
how the perturbations in the parameters of a previously trained implicit-based
model alter the shape boundary. The authors consider two types of editing
operations: (i) geometric editing, which applies a geometric update on the
entire shape boundary and (ii) semantic editing, which applies a target
displacement only to a part of the boundary. The paper does not compare to any
baseline, nor reports any quantitative metrics to evaluate the quality of the
editing operations.
strength_and_weaknesses: ## Strengths:
-------------

1. I really like the fact that the proposed method is model-agnostic and can be
applied to a pre-trained Neural Network to deform a shape in place.

2. The idea of employing boundary sensitivity to associate changes in the
parameters of a previously trained model with the implicit shape is novel and
has many potential applications to various tasks.

3. The proposed idea is very simple and potentially can be applied for editing
various implicit-based shapes.


## Weaknesses:
-------------

1. While I really like the idea of this paper, unfortunately the experimental
evaluation is quite weak. No baselines are considered and no quantitative
metrics are reported, hence making it quite hard to evaluate of final edits.
While qualitatively the results are very impressive, I believe it is necessary
to also include some quantitative metrics that measure the realism of the final
shape. One potential baseline to consider, which is not 100% comparable with
the proposed method is the Deformation Handles. While Deformation Handles
cannnot operate on the implicit shape it might be interesting to see whether
they can yield similar results when applied to the reconstructed shape

2. Various details from the Applications section are missing. For example, in
section 4.1, the authors state "The considered models are trained to fit the
SDF and surface normals a shape and share the same architecture: sin activation
functions and 3 hidden layers of 32 neurons each." This sentence is a bit
unclear and contains some typos so I am not really sure what is the
experimental setup in this section. Do the authors train their model on each one of
the shapes in Figure 1? What is the input? How are the displacements
parametrized? Αre they defined only on the normal direction?

3. Another point that is a bit unclear from the text is whether the editing
operations are defined in relation to some target shape. I think that the
authors only define a target deformation rather than a target shape. Can the
authors please clarify this?

4. Another limitation of this work is that the authors do not experiment with
various network architectures. I think it would be useful to showcase that
their model can be applied on various implicit-based architectures such as
OccNet, DeepSDF, NeUS etc.

clarity,_quality,_novelty_and_reproducibility: While I really like the idea behind this work and I think it could have been an
amazing paper, I think the writing of the paper could be improved, since in its
current state many implementation details are not properly discussed.
While the authors do not introduce a novel methodology, the idea of using
boundary sensitivity to edit implicit shapes is novel, to the best of my
knowledge.
summary_of_the_review: While I really like the idea of this paper, I am a bit concerned regarding the experimental evaluation, since the authors do not compare to other methods nor report any quantitative metrics. Moreover, since various details are unclear I am leaning towards rejecting the paper. Next, I have included various comments and questions regarding the paper:

1. In the 4th paragraph of Section 1.0, the authors state "the user supplies a
target displacement on (a part) of the shape boundary". This sentence is a bit
unclear. How is the target displacement defined/parametrized? While additional
details are provided later in the text, I think it is important to add one/two
more sentences at this point to elaborate more on this.

2. The Related Work section can be improved. For the first two groups of
papers, i.e. Implicit Shape Representations and Manipulation and Neural Fields,
I think it would be nice to clearly state that these works are not related work to the
proposed method. Instead the proposed method can edit implicit shapes generated
from these models by applying perturbations on the NN parameters. Regarding the
Neural Shape Manipulation section, I believe it would helpful to clearly
explain how boundary sensitivity is deployed in the works of Atzmon et al.,
Remelli et al. and Mehta et al., as stated in the last sentence of this
section. This analysis is important in order to better understand what is novel
in this work and what was already possible in prior research.

3. For the bottom right experiment in Figure 1, what is the editing operation
that is considered? Moreover, I don't fully understand why the ABC letters
disappear?

4. I think that the experiment in Figure 2, is not properly discussed. Can the
authors explain what was the message they intended to convey with this
experiment? 

5. I think it would be great if the authors could also show results on the
geometric shape editing task on the IM-NET model that is used in the
experiments of Section 4.2. Being able to edit implicit shapes of a model
trained on multiple objects is a more challenging setup.


Typos:
-----
- Caption in Figure 2 has typo: depthg -> depth
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This submission proposes a mathematical framework for editing 3D represented as deep implicit surfaces. Given a target displacement on the shape, the corresponding modifications of network parameters is computed and applied.
This can either be applied to a network representing a single shape (shape editing) or to the latent code of a multi-shape network. In the latter case, the network acts as a learned prior and allows semantic editing.

strength_and_weaknesses: The theory supporting the framework is very sound, clear, and easy to follow. As opposed to previous works deriving similar formulas ([1] Controlling Neural level sets, Atzmon et al., NeurIPS 2019 and [2] MeshSDF, Remelli et al., NeurIPS 2020), authors here explicitly write the the tangential component before zeroing it out.

As experiments demonstrate, it effectively works for shape modifications.
The provided visualization of “basis functions” (ie. ~gradients at intermediate layers) are nice and shed a light on what the network has learned, but cannot be used for any practical purpose.


My main concern is that this whole idea of finding gradients for network parameters corresponding to some shape modifications is not novel. [1] and [2] derived the exct same gradient, and [3] Sketch2Mesh, Guillard et al., ICCV 2020 already used it for interactive shape manipulation. The proposed method loses some generality compared to these frameworks which express gradients and chain rule for any downstream gradient, not only displacements.

The only 2 minor differences compared to [1] and [2] are:
- 1: [1] and [2] use gradient descent to update parameters, here a linear least square is used.
- 2: [1] and [2[ always zero out the tangential component of the gradient, here it can be non zero.

To convince me of the relevance of their approach over [1,2], authors would need to argue and demonstrate at least one advantage. It could be in terms of:
- Speed? Would linear least square faster than gradient descent?
- Using a non zero tangential component? This is only used in one experiment, not very well explained (I could not reproduce), and not ablated.
- Not having to reconstruct the explicit surface? But I think this is a misleading argument, because to pick a place where displacements must be applied, one must get an explicit representation of the surface first. And the reprojection of points sounds very much like an adaptive remeshing algorithm. Even more so if that was true, what would be the point of avoiding an explicit mesh in the loop? Speed? Marching Cubes is very fast with coarse to fine grids. Precision? Marching Cubes precision far exceeds the accuracy of the upstream MLP network…

Another minor concern is that it is not very clear to me how the displacement is indicated on the shape.

clarity,_quality,_novelty_and_reproducibility: Very clear paper, reproducible (apart for experiment 4.3 involving the tangential component as mentioned above). However, not much novelty.
summary_of_the_review: Nicely executed paper, but I don’t see the added value compared to ([1] Controlling Neural level sets, Atzmon et al., NeurIPS 2019 and [2] MeshSDF, Remelli et al., NeurIPS 2020) or even [3] Sketch2Mesh, Guillard et al., ICCV 2020
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method for interactively editing shapes represented as level sets of neural implicit functions.

Strengths:
- Interesting method for computing boundary sensitivity of an implicit field
- Supported by sound theory
- Various applications presented to demonstrate usefulness of the approach
- Is (in principle) model-agnostic; can be applied to different neural field architectures

Weaknesses:
- No discussion of weak points / failure cases
- Doesn't really compare to any plausible baselines
- Model-agnosticism is not evaluated very much (only a couple of architectures are tested, each on a different application, so there's no side-by-side comparison).
- Authors missed a couple of very relative pieces of prior work that do something similar. To my eye, this is the most serious criticism raised by any reviewer, but this reviewer is actually currently slightly in favor of accepting the paper.

Reviewers did engage in discussion with authors during the discussion period. Still, there was no clear signal on whether reviewers were generally in favor of accepting or rejecting the paper, so I called for a meeting.
justification_for_why_not_higher_score: Given that I (and reviewers) am conflicted about whether the paper should be accepted at all, I have a hard time justifying a higher score than a poster accept.
justification_for_why_not_lower_score: The paper offers enough value to the research community that I can't say with confidence that it should be rejected.
Decision: Accept: poster