Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper extends the NeuralCages work to NeRF to enable the deformation of implicit fields. Given a NeRF, it first extracts surface points and then optimizes a template cage to enclose the points by penalizing the negative MVC values of the sampled points. It then follows NeuralCages to deform the cage. All points in the cage can be deformed accordingly with barycentric coordinates and the vertex position of the deformed cage. The author proposes a sampling trick to handle the aliasing issue (I don't understand this part).

In general, the idea of combing NeRF and NeuralCages is straightforward. However, the presentation is far from satisfactory. The experimental results are not convincing or impressive.
strengths_and_weaknesses: Strengths:
The idea of introducing cage deformation to NeRF is interesting. The overall idea of combing NeRF and NeuralCages is straightforward and technically sound.

Weaknesses:
1. Although the overall idea is simple, the presentation is far from satisfactory and hard to follow. Especially, the terms and symbol system are complicated and confusing:

(a) What do the terms `geometry` and `geometry proxy` refer to? Are they refer to surface meshes only, surface points, or the regions/points within surface meshes?

(b) geometry \mathcal{G}_{can} is a set of surface points? but the geometry proxy \mathcal{G}_{p}(\mathbf{v}, \mathbf{f})  is a mesh?  What about deformed geometry \mathcal{G}_{n}? Again, the meaning of symbel \mathcal{G} is confusing. 

(c) If you want to say a mesh M with vertices V and faces F, please use M={V, F}. M(V, F) seems like a function taking two arguments. For example, the text uses \mathcal{G}_{p}(\mathbf{v}, \mathbf{f}), which is strange.

(d) Line 132: which nearest neighbor algorithm? Please add a reference. Does it generate a watertight manifold mesh?

(e) Line 139: sample points from mesh surfaces or the interior. If sample points from the interior, why \mathcal{G}_{p} is closed mesh? If sample points from surfaces, can we directly sample from \mathcal{G}_{can}? And there is no need to construct \mathcal{G}_{p}?

(f) Equation 5: does \mathbf{v} refer to a sampled point? If so, please use another symbol instead of \mathbf{v}, which refers to vertices before. Also, I cannot understand why it is ``max min``? We need to penalize all negative MVC values, and it should be ``min min`` instead?

(g) Line 157: what is \mathbf{E}_{C}?

(h) For the deformed geometry \mathcal{G}_{n}, what does `n' refer to? It's kind of confusing.

(i) Line 219-220 says \mathcal{G}_{s} and \mathcal{G}_{p}. But in the caption of Figure 4, it says \mathcal{G}_{s} and \mathcal{G}_{n}. Which one is correct?

(j) Equation (14) what does `g' in \mathbf{p}_{n}^{g} refer to?

(k) Do we really need \mathcal{C}(\cdot)? Only mentioned once.

In general, section 3.2 - 3.4 is very confusing and wordy. I tried hard to follow but still failed to follow all the contents. I suggest the authors rewrite it more precisely, clearly, and compactly. A figure to illustrate the relations between the symbols may also help.

2. In section 3.3, the network architecture, loss functions, and training details are missing.

3. I can roughly understand the reason for the aliasing issue. But why do we need a vicinity threshold \epsilon? If \epsilon is very small, do we still have the aliasing issue? Moreover, I have no idea of how and why it is addressed by introducing a shell \mathcal{G}_{s}. Motivations and discussions are missing.

4. The setting of ``Synthesized Object Animation'' is kind of confusing. If we already have a sequence of animated meshes, we can directly render the images. Why do we still need the proposed CageNeRF method? I am also confused about how the method actually works (Line 256-258).

5. For Pose Reenactment, do we have meshes of humanoid objects? If so, can we directly apply NeuralCages? Why do we need CageNeRF?

6. Both Table 1 and Table 3 are computed over a single shape? What about Table 2? The experiment results are not convincing or impressive.

questions: See weaknesses. 
limitations: The authors have mentioned some limitations in supplementary materials.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a way to deform Neural Radiance Fields using a cage-based paradigm. It is based on the training of a cage deformer, optimized on the poses. Then, given a new pose, the method regresses the cage on an explicit geometry and computes a dense mean-value coordinate field to compute the new visualization. The method is tested on articulated (i.e., humanoids) and non-articulated (i.e., chairs) objects.
strengths_and_weaknesses: == Strenghts ==

- The approach seems novel and addresses a complex problem. Providing novel tools to edit Neural Radiance Fields is a hot topic with a potentially great interest for the computer graphics community
- The method seems applicable in different contexts, with good results both for rigid objects and non-rigid objects
- The method is well discussed in its elements, providing ablation and justifying the design choices.

== Weaknesses ==
- Looking at the video, there seem to be some hole artefacts, e.g., under the lifted arm of Hulk, in the pose reenactment experiment. Is it due to the stretching introduced by the paradigm, or is it because the NERF itself does not well represent some areas?
- The method still requires access to an explicit geometry, which somehow limits the arbitrary resolution power typical of neural representations. When an SDF is used, it requires a second training for the representation. 
- The shown animations are not particularly complicated; probably, the use of cages requires to limit of the intersections, but I cannot understand if the method is applicable also in more complex scenarios with more articulated movements.


== minor fixes ==
- Line 174: missing a comma after "$o$ is the camera origin"
- Line 291: typo, "Impact"

== After Rebuttal ==

After the rebuttal and the discussion phase, I am prone to keep my initial rating and vote for acceptance. For the final version, I suggest including in the main manuscript the observations discussed in the reviewers-authors discussion, especially about the limitations and trade-offs of the method. 
questions: - Why do some of the results present hole artefacts?
- Why the method is not tested on more complex movements? There is any limitation to taking them into account?
limitations: The paper fairly discusses limitations and societal impact, while on the first probably a bit more elaboration on the kind of animations admitted by the method can be discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: CageNeRF allows for deforming a neural radiance field of arbitrary surfaces/objects, unlike prior work that was restricted to specific categories like humans or faces by design. The deformation network is trained per object and requires training data for deformations. The method design uses an invertible/bi-directional deformation parametrization based on classic graphics knowledge, namely deformation cages. It is an extension of the prior work Yifan et al. (Neural Cages for Detail-Preserving 3D Deformations; CVPR'20) to implicit fields. This extension is non-trivial due to the volumetric and non-discretized nature of the neural radiance field. The method is evaluated on deformation transfer, reenactment, and animation. All these applications apply a deformation cage to a neural radiance field, with the deformations coming from the same sequence as the radiance field in the case of synthetic reenactment and from a different sequence/scene in all other cases.
strengths_and_weaknesses: Strengths:
* Editing the neural radiance field of arbitrary surfaces/objects is an important direction for the NeRF field.
* CageNeRF is a fairly elegant parametrization and tackles e.g. aliasing issues well. 
* Only human reenactment allows for a comparison against prior work, namely AniNeRF for human-specific NeRF animation. CageNeRF performs worse but close enough in my opinion, considering that it is a general method.
* The paper is written in good English.

Weaknesses:
* The method section skips over non-obvious steps in many places, see below in Questions.
* Deformation transfer is demonstrated only qualitatively and only on the ShapeNet chair category for a handful of (random) instances. In addition, this leads to deformations that seem almost just like per-axis scalings. Other categories with more interesting deformations would show that what the method does goes beyond what an anisotropic matching of the object bounding boxes could do. I don't believe the deformation transfer results are sufficient evidence that deformation transfer actually works with the method.
* The quantitative results in Table 1 are on 3-4 specific frame indices out of dozens of indices per sequence (on top of evaluating only two sequences). That leaves the door wide open for cherry-picking a few good indices. Averages (and standard deviations) across the entire sequences should be reported. 
* The deformed cage is determined by a neural network and hence might not generalize well to deformed target poses that lie far away from the training data.
questions: * An argument why a comparison against face-specific NeRFs is not necessary would be good.
* What is the speed of the method? By what factor does the rendering slow down due to the deformations, i.e. relative to just directly using the canonical NeRF, skipping C in Eq. 15?
* I don't understand how the grid sampling (L195) interacts with the volume rendering. The points p from the rendering are first along a straight ray. How are these then transformed via the cage into the canonical radiance field? The points p won't lie on a grid but anywhere in space, so how does the grid come into play? I understand the thresholding with epsilon.

Minor questions that are relevant for a revision but not the rebuttal:
* Eq. 5 is not easy to parse. "v" is used for the vertices of the cage template, the proxy geometry, and as any individual vertex of the proxy geometry. I cannot tell which of these "v" the "max" is over. I believe it should be the vertices of the proxy geometry.
* I also don't understand how this is turned into a neural-network optimization problem. What is the loss? If this information can be found in Neural Cage, I strongly recommend to add a Background section explaining the important parts of Neural Cage that are used by this submission.
* How is the cage deformer applied in Sec. 3.2 (L137f)? Is C_t the "Optimized Cage", G_p the "Novel Pose", and C_can the "Deformed Cage"? If so, this is not at all obvious and needs to be stated explicitly.
* In Sec. 3.3, what is the loss, during pretraining and for Sec. 3.2? 
* In Fig. 3, both the lower and upper branch use E_C to my understanding. It would be helpful to the reader to visualize both in the same way, e.g. use the "E_C" block also for the upper branch instead of the words "Feature Extraction".
* In Sec. 3.3, please mention that a global PointNet++ encoder is used. It was not clear to me for a while whether the deformation might be regressed per-vertex (coordinate-based MLP) instead for all vertices at once. The supplement also gives the unclear description "to predict the vertex-wise offset for the input cage base" (L85). 
* In L216ff, G_d, G_p and G_s are used incorrectly, I believe. G_d should be G_n at the very least.
* Are the numbers in Table 1 from rendering into a single view or multiple views per frame index?

Typos etc.:
* L170: Not a full sentence.
* L291: Impact
limitations: Limitations are mentioned but I believe that it should be also explicitly mentioned that the cage deformer might not generalize well to target poses outside the training distribution since there is no evidence to the contrary in the submission.

General broader impact is discussed but no potential negative societal impacts are mentioned. Deformation transfer could have the typical deep fakes issue where the identity of one person is animated by motions of another.
review_title: null

Meta-review: metareview: All reviewers consider the central idea of the paper to be novel and interesting, and that deformable NeRFs are a valuable research area.

All reviewers agree that the initial paper was poorly presented, and that the revised version is considerably improved.

While accepting the authors' rebuttal regarding the lack of suitable datasets to evaluate deformable NeRFs. the qualitative presentation could be improved further, and in particular, it is recommended to include video sequences of continuous deformation, which will more clearly show the artifacts of both the proposed and other methods.

confidence: Less certain
Decision: Accept