CageNeRF: Cage-based Neural Radiance Field for Generalized 3D Deformation and Animation
Authors: Yicong Peng
Yichao Yan
Shengqi Liu
Yuhao Cheng
Shanyan Guan
Bowen Pan
Guangtao Zhai
Xiaokang Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural Radiance Field
3D Reconstruction
Cage Deformation

Abstract: While implicit representations have achieved high-fidelity results in 3D rendering, it remains challenging to deforming and animating the implicit field. Existing works typically leverage data-dependent models as deformation priors, such as SMPL for human body animation. However, this dependency on category-specific priors limits them to generalize to other objects. To solve this problem, we propose a novel framework for deforming and animating the neural radiance field learned on \textit{arbitrary} objects. The key insight is that we introduce a cage-based representation as deformation prior, which is category-agnostic. Specifically, the deformation is performed based on an enclosing polygon mesh with sparsely defined vertices called \textit{cage} inside the rendering space, where each point is projected into a novel position based on the barycentric interpolation of the deformed cage vertices. In this way, we transform the cage into a generalized constraint, which is able to deform and animate arbitrary target objects while preserving geometry details. Based on extensive experiments, we demonstrate the effectiveness of our framework in the task of geometry editing, object animation and deformation transfer.

CageNeRF: Cage-based Neural Radiance Field for Generalized 3D Deformation and Animation
Yicong Peng'
Yichao Yan
Shenqi Liu?
Yuhao Cheng2
Shanyan Guan
Bowen Pan
Guangtao Zhai
Xiaokang Yang²
1 Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University
MoE Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University
3 Alibaba Group
(jack- pood--pparoo.yyychoaaann. chengyuhao, shyanguan, zhai guangtao suanggaa,xkyang)6Ba. xkyang)@ajtu.. bowen, bbwb.:ppaaaaaabbbbbbbbbbbbab
zhaiguangta
edu.
Abstract
While implicit representations have achieved high-fidelity results 3D rendering. it remains challenging to deforming and animating the implicit field. Existing works typically leverage data-dependent models as deformation priors, such as -ppegory-specific
SMPL for human body animation. However, this dependency priors limits them to generalize to other objects. To solve this problem, we propose novel framework for deforming and animating the neural radiance field learned arbitrary objects. The key insight is that we introduce a cage- based representation deformation prior, which is antegory-agnostie is performed based on an enclosing polygon mesh with sparsely defined vertices called cage inside the rendering space, where each point is projected into novel position based on the barycentric interpolation of the deformed cage vertices. In this way, we transform the cage into generalized constraint, which is able deform and animate arbitrary target objects while preserving geometry details. Based on extensive experiments,
Specifically, the deformation
demonstrate the effectiveness of our framework in the task of geometry editing, object animation and deformation transfer. The code and supplementary materials are available https://pengyiconn github.io/CagNeRR/
Introduction
Editing and manipulating 3D object is a fundamental task for generating immerse contents, which enables numerous applications in VR/AR. Although recent advances in implicit neural rendering, such as neural radiance field (NeRF) [29], are able to produce photo-realistic modeling of static objects, how edit the 3D content remains challenging task. In this paper, focus on learning general deformation field for NeRF, enabling the deformation and animation of general objects.
Editing explicit 3D representations, such as point cloud and mesh, has been well studied. Point cloud and deformable mesh [6] [3] store the geometry information in 3D space, which are directly editable and widely utilized in computer graphics for modeling dynamic objects (30). Case-based deformation 22 21 25,, for example, has been used to animate polygon mesh, where per-vertex coarse cage. While mesh and point cloud representations are relatively flexible to edit and animate, it difficult to produce geometrically accurate and fisually-faithfu complex objects and scenes. Additional data such as normal maps and textures are usually required to achieve high-quality rendering results.
deformation be generated by translations prescribed
Recent works on implicit neural rendering, such as NeRF, employ multi-layer perceptron (MLP) as a universal approximator [5 37 allowing them to approximate a continuous volumetric scene
"Corresponding authors
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Multiview Inputs
CageNeRF Outputs
Geometry Editing
Object Animation
Figure Given multiview images of an object, our method learns an implicit deformable 3D)
representation, which is used to deform and animate the object.
function to reconstruct high-fidelity results with only sparsely sampled images. Therefore, it is highly desirable if the neural rendering models can be also applied to model dynamic objects. However, optimizing a deformable radiance field end-to-end for dynamic objects like human body highly under-constrained problem. To address this issue, when modeling dynamic objects, most works optimize inside deformation-invant
canonical space with data-dependent priors, such as
3DMM 4 for face deformation [1], and SMPI. (26) for human animation [34] 32). While using data-dependent priors when modeling specific dynamic objects proved to be effective in providing
stable correspondence between the object structure and the implicit field, it also limits the model's
capability to generalize for broader applications.
To address this issue, we propose a novel framework which focuses on deforming and animating general objects represented by neural radiance fields, as shown in Figure []] We take inspiration from the vanilla cage deforming approaches 200 47] and extend them to work with implicit neural radiance fields. However, directly applying cage deformation into implicit field is non-trivial, due to the following issues. i) Traditional cage deformation methods work on polygon mesh with finite vertices and faces, while implicit radiance field continuous 3D volume. ii) Cage-based deformation transfer requires explicit geometry of the target, which is not directly accessible in the implicit field. iii) It is computationally expensive to densely sample and discretize the implicit field. To address these issues, we propose to estimate a coarse geometry of the target and generate discrete field which supports bi-directional mapping between the canonical space and the deformed space, while keeping the radiance field continuous. Moreover, further design novel sampling strategy to reduce the computational cost while maintaining the render quality.
The advantages of our framework are three-fold. First, by utilizing category-agnostee cage-based deformation, our framework does not rely on data-dependent priors and can generalized to work with a wide range of objects. Second, since cage-based deformation is self-contained method, once the model is trained on the static scene, no additional training is required to apply deformation. Third, both cage deformation and volumetric rendering operate on the Euclidean space, we can design our framework in a decoupled manner as long as we keep the underlying coordinate system consistent between the two modules. This increases the flexibility and versatility of our framework. Moreover, as well-developed method in computer graphics, cage- based deformation can be used in various tasks, which further expands the application scenario of our framework. We demonstrate the effectiveness of our framework geometry editing, object animation, and deformation transfer.
2 Related Work
Novel View Synthesis. Traditional methods for novel view synthesis rely explicit representations of 3D scenes. In these methods, 3D objects are modeled by meshes, point clouds [50] 38), etc. and then novel views are generated by shading and rendering. However, high-fidelity textures and well-designed 3D models are required for photo-realistic rendering, which makes the procedure highly expensive and inefficient. Recently, NeRF 22 has achieved impressive performance on novel view synthesis, where 3D objects are represented by implicit functions which map positions to densities and colors. A number of follow-up works have made further improvements to NeRF

---

Multi-view Input Images
Neural Renderer
Canonical Radiance Field
Edy (p: s.))
(x,).2)
Novel View (Canonical)
Cage Deformer
Cage Deformer
Eeff (oy vt)
Optimized Cage
Deformed Cage
Deformed Radiance Field
MLP
Novel Pose (Deformed)
Novel
Overview of CageNeRF. CageNeRF consists of two modules: (1) a neural rendering module, which learns a radiance field from multi-view input images in the canonical space; (2) a cage deformer module that produces general cage deformation to manipulate the learned radiance field. CageNeRF deforms the implicit radiance field in end-to-end manner.
Figure 2:
Although these NeRF-based implicit methods are good representing static scenes, it is still challenging to model dynamic scenes.
to yield synthesis results with higher quality 53
52 45
Explicit Geometry Deformation. Modeling deformable 3D objects a fundamental task in computer graphics and computer vision, and triangle meshes are most commonly utilized due to their flexibility manipulation. Traditional methods achieve this by deforming fixed set of vertex points and then interpolating values among them [22 46), such skinning on skeletons 17
28 or cages- based
methods 21 40 [6]. Along with the advances of large-scale datasets S
7
learning-bai based
methods are proposed by analyzing the prior principle of mesh deformation 9 47
J0 19]. Wang et
al. [47] proposed a differentiable cage-based deformation module to synthesize shape variations and perform deformation transfer. However, these deformation methods can be only applied to explicit models, e.g., meshes. It is expensive to generate high-fidelity textures for photo-realistic rendering, and these methods cannot be directly applied to implicit models.
Implicit Geometry Deformation. Compared explicit representations that only discretely control the vertex of meshes, the deformation of implicit geometry is more challenging. since the space is represented with a continuous function. Recently, various approaches build different deformation fields on implicit representations for animation B1 [8
However, the generalization abilities of these methods are limited with respect to novel poses and different objects. Referring to the widely concerned digital humans, a vast range of methods are proposed for the deformation of the implicit
42 43
face 12 57 14] and body 334 36 33). However, these methods heavily rely on parametric models as priors, such as 3DMM 4 and SMPL (26). which are difficult to transfer to non-humanoid objects. this paper, we propose a cage-based based implicit geometry deformation method, which utilizes sparse neural cage as guidance build the deformation field for NeRF. The proposed CageNeRF is a general algorithm which is applicable to a wide variety objects.
3 Methodology
We propose a novel framework. dubbed CageNeRF. for editing and animating arbitrary 3D objects represented by a neural radiance field. Radiance fields can render photo-realistic content. However, remains challenging to perform deformation on radiance fields, since it's non-trivial learn correspondence between implicit representations. We address this challenge by employing the cage-base deformation, which offers an elegant way of manipulating points inside continuous space through a predefined coarse triangle mesh called cage. As shown in Figure our framework consists of two modules: the neural rendering module optimizes the radiance field of a given static object inside canonical space using multiview images with known camera parameters. Then the cage deformer module deforms the cage inside this canonical space and reconstructs the implicit radiance field by translating points to novel positions.
3

---

3.1 Canonical Space Neural Rendering
We first train our neural renderer module to acquire the NeRF volume [29]. which acts as a reference canonical space. NeRF models the scene along with its geometry. lighting, and material as continuous volume and approximates it as a radiance field using an MIP network. Given a 3D position of point (z; y, z), the network learns a mapping function of its density and color c (T' g, b) variations with respect to the view direction d ():
Fe (r((), 7(d) - (c,o),
where 7(.) is a positional embedding function of sinusoidal basis and θ is the trainable parameters. Then, volumetric rendering is performed to synthesize images by marching camera rays r + id from viewpoint o, with direction d, through the radiance field. The pixel color C' is estimated using discrete integration on points sampled along each ray:
C(r) Σ exp( oo - -xx(-o)))p)-
where 5; = 44+1 & is the interval between adjacent sample points. We then optimize our neural render module inside this canonical space using L2 loss calculated between the estimated color C(r) and ground truth pixel value C(r) of the input images:
C= Σ [|(())) C(r)112
(3)
FER
where R denotes the set of rays that contributes to sampled pixel.
3.2
Cage Optimization
To deform the neural radiance field, we need to establish a consistent correspondence between the canonical and the deformed fields. To achieve this goal, a cage, i.e., coarse polygon mesh inside the canonical space is first optimized to geometrically aligned and enclose the object. Then the deformation field can be interpreted as the deformation based on the structural constraint of this optimized cage. Similar Neural Cages, we provide our network with predefined polygon meshes as the starting point of this optimization process.
The geometry the object inside the canonical space is represented by a polygon mesh G= {v, f). where v and f represent the vertex and face, respectively. When an explicit surface is not available, G can be obtained by applying the marching cube algorithm (27 on radiance field SDF 45]
The template cage C is a polygon mesh containing N vertices, which is optimized w.r.t. the mean value coordinates (MVC) of sampled vertices of G. Since negative MVC occurs when 1) the point lies outside the cage and 2) the cage itself self-overlapping or highly concave. By penalizing the negative values, this optimization process automatically drives the vertices of the template cage enclose the object inside the canonical space:
min min (Ø: (v), 0)1 E 9, 1,2, N,
where (be()) is the MVC weight function w.r.t. sampled vertex from polygon mesh and the i-th vertex on cage C.
3.3 Cage-based Deformation
After the cage C is optimized w.r.t. an object, we can perform deformation based on the target novel pose/shape and the current cage shape. While it is possible to perform the cage-based deformation by explicitly editing the cage vertices, we take inspiration from Neural Cage [47] and design a cage deformer module, which generates deformed cage based on novel polygon mesh Gn, given as the deformation target. This cage deformer module makes our framework applicable to tasks like deformation transfer and object reenactment. Similar to the network proposed in Neural Cages, this module takes an encoder-decoder structure with PointNet++ (35) as its backbone to automatically calculate the displacement prescribed on each vertex of the optimized cage C, shown in Figure

---

We pretrain cage deformer in two different datasets: 1) ShapeNet 77 for deformation trans- fer and geometry editing on general objects; 2) an extended version of Surreal dataset [T]] for
E
f
Optimiaad Capa
better generalization ability on novel pose syn- thesis and animation.
E
Defored
When performing the cage deformation, the coder E extracts geometry features from the optimized cage and the given deformation tar- get Gr. and outputs Se and /t. respectively. The decoder D takes in the concatenated features and outputs the displacement Av; for each vertex cage C:
Novll Pose
Figure 3: Structure of the cage deformer.
(5) where L refers vector concatenation and Áv the per-vertex displacement w.r.t cage C. Since the cage deformation only changes vertex position while preserving topology, the deformed cage C can be obtained by adding the displacement Av:
Av = D(So + fi), E RN33
C' = (v + Av,
(6)
The deformed cage C' is later used to generate an MVC field for radiance field editing.
3.4 Radiance Field Editing
Once the deformed cage C' is acquired, we can edit the radiance field with the following two steps. 1) Deforming the original geometry G acquired in Section 3.2 into a novel mesh g' using traditional rage-based deformation. This newly generated mesh is used as the 3D mask to filter out redundant sample points and speed up the rendering process. 2) Building a dense MVC field, which acts as the consistent correspondence during deformation, based on the deformed cage through grid sampling.
Deformed mesh as 3D Mask. The deformed mesh G' can obtained by performing cage-based deformation directly on the extracted mesh G of the object inside the canonical space. For every vertex V, E G, we first express its position using MVC coordinates:
v; = C(vi) = Σ (w
where (-) series of weight functions depending on the relative position between the cage vertices Vj and the given mesh vertex v,. The detailed derivation of the 3D barycentric interpolation given in Appendix ??. Then, the vertices of the deformed mesh G' can be obtained by:
V, Σ SivVv V'y' 5=1
(8)
where every vertex v; on G' has novel position determined by the deformed cage vertices v c' as its new basis.
Since radiance field is sparsely occupied by the render target, in order to increase the rendering speed while maintaining visual quality, we use the deformed mesh G' 3D mask, where only the points within the vicinity threshold of the mesh surface are sampled during volumetric rendering.
Moreover, this explicit mesh enables us robustly establish the backward deformation, i.e., from the deformed space to canonical space, when canonical information is not present in datasets like Human 3.6M [16]. See appendix for detailed discussion.
Anti-aliasing MVC Sampling. As shown in Equations 78 while the coordinate representation of point changed in Euclidean space, the MVC representation remains invariant. We extend this
5

---

property on meshes into 3D volumes, which yields our editable radiance field. By utilizing this invariant correspondence, we are able effectively edit the radiance field via volumetric cage-based deformation:
Fa= (p= (5.. & O Ip E F):
where every point p in this deformable radiance field Fa has its MVC coordinates calculated under a given cage C. F is the canonical radiance field learned by our neural renderer. And the deformation in this neural volume can be carried out by replacing the original cage C with a novel one C'.
Since the MVC values remain invariant once the cage is fixed, to reduce computation cost, we build pre- sampled MVC field based on the deformed mesh g' as is shown in Figure E Specifically, we apply a
n
MVC Field
72
grid sampling strategy the deformed space, and the MVC of the sampled grid point P, is equal to the barycentric coordinates of the closest point on the deformed mesh Pn E G to Pg.
MVC Aliasing
P
Po
71
While this sampling strategy is efficient, it causes the aliasing issue shown in Figure I The points (Po- PI, P2) are sampled grid points, since they all project to the same closest point Pa on the de- formed mesh g', i.e., they share the same MVC value. When rendering the deformed object, sampled points {Po. P1, P2} are projected to the same position in- side the canonical space and thus have the same color and density value. Aliased sampled points such as P2 have the same color and density as Рn. instead of the correct transparent color. This causes the "ex- panding" effect in the deformation space, e.g., the deformed object in the figure will be rendered as a circle with radius r + e, which should be instead. One naive solution would be directly reducing the vicinity threshold This approach imposes a higher demand on the accuracy of both the implicit repre- sentation and the extracted polygon mesh which is non-trin non-trivial. To mitigate this issue while maintaining a relatively large we additionally define shell Go. which is also a polygon mesh that closely bounds the surface of g' by small gap. This shell can be obtained by translating vertices on G' along their sur- face normal while maintaining the topology. When assigning MVC value to grid points, we compare its distance to both G. and g' and determine the MVC value according to:
O
6'
gs
E
Figure 4: Illustration of anti-aliasing MVC sample strategy in 2D case. Grid points projected onto the target geometry w.r.t. the surface normal 7.. Given sample range e, Points 10.1.2 among have the same nearest point Pa. on deformed mesh g' and thus share the same MVC value. This causes aliasing when rendering the deformed radiance field. Note that Grid points outside are filtered out. Considering this problem, we devise shell G- (dotted blue circle) which closely bounds the render geometry by small margin. Then, we assign the MVC value by comparing the distance from the grid point both the de- formed mesh g' and the shell G. negate this effect.
{ Og: |Porid pyll2 S |Parid - P=112 do. [Parid p, > |Pyrid P=ll2
Dgrid =
(10)
where Øyrid is the MVC of the sampled grid point, dy the MVC of the closest point Py on mesh g', d. is the MVC of the closest point p. on shell G.. When rendering the deformed neural volume,
points such as P1.2 with MVC of de are projected outside the object in canonical space and thus have correct transparent color.
Deformed Rendering. After acquiring this MVC field, can render the deformed radiance field through volumetric accumulation. 1) Sample points along the camera ray in render space using the deformed mesh g' as a 3D mask. 2) Calculate the MVC value of filtered points using the MVC field via trilinear interpolation based on their coordinates. 3) Apply the cage deformation using the sampled MVC value to project points back to canonical space. 4) Infer the color and density of sampled points inside canonical space and apply volumetric rendering:
F8 (x(C(p)), (d)) + (c.
(1I)

---

Source
Deformation Targets
camnonical
scale
Different Views
rotate
Figure 6: Results of explicit geom- etry editing. Top row shows the original object rendered in canon- ical space. Row 2-3: deformed model under different views.
Figure 5: Results of geometry editing via deformation trans- fer. Top shows the source object with texture and defor- mation target geometries. Row 2-3: deformed model under different views.
Table I: Animation Performance Synthetic Datasets
Dataset
Hulk
Whale
Frame Index
27
47
Avg.
100
Avg.
PSNR SSIM LPIPS
36.434
35.327
34.078
35,466
36. 346
35.723
35.755
35,085
35.742
0.963
0.957
0.948
0.959
0.972
0.972
0.971
0.966
0.972
0.0126
0.0156
0.0188
0.0153
0.0117
0.0121
0.0119
0.0126
0.0120
where C(-) is the cage deformation that translates sampled points back to canonical space.
Experiments
To verify the generalization of cage deformation, we evaluate CageNeRF on three tasks:
Geometry editing (Section 4.1). In this task, we demonstrate the capability of CageNeRF in editing the geometry of the implicit object by deforming the radiance field. This is achieved by either explicitly editing the cage shape or using the cage deformer module generate novel cage via deformation transfer. Deformation transfer is a task that takes two objects with different geometry properties (e.g., shape, structure and scale), and modifies these properties on the source make resemble the target. In this task, CageNeRF is trained on the lego dataset used in NeRF (2 and chair dataset generated in the same manner.
Neural animation (Section 4.2). In this task, we evaluate the continuous deformation quality of CageNeRF by comparing the results with targets animated using Linear Blend Skinning (23 We both synthesized objects and human videos as our henchmarks. The synthesized 3D objects contain two animated characters: 'Hulk' and blue whale. Furthermore, we evaluate CageNeRF on real humans using Human 3.6M [16].
Pose reenactment (Section 4.3). In this task, we generate synthetic dataset using static model of a humanoid robot, which only contains polygon mesh and texture without rigged animation. This dataset contains the rendered 360 multi-view images with paired camera parameters. We randomly sample posing SMPL models from the Surreal dataset 44 as the reenactment target.
We further provide the ablation study w.r.t. the MVC sampling strategy. Following the majority of previous literature [29 32 45), we use PSNR/SSIM [48 and I.IPS (44) as evaluation metrics.
We refer readers Appendix ?? for more details and results.

---

LBS
Driven
Ground Truth
Ours
AniNeRF
LBS Driven
Ours
Ours
Figure 7: Qualitative results of neural animation on Whale and Hulk datasets. Row 1, 3: driven by linear blend skinning. Row 2, 4: driven by our framework.
Figure 8: Qualitative compar- ison of CageNeRF and AniN- eRF [32 for human synthesis Human 3.6M.
4.1 Geometry Editing
We demonstrate the effectiveness of deforming raidance field via explicit cage editing on the lego dataset as shown in Figure 6 We edit the cage geometry to achieve the effect of stretching and rotating the bucket. In our framework, deformation transfer can be applied to edit the geometric properties of the implicit 3D objects. Our framework is first trained on an office chair in the canonical space, as a deformation source. We randomly select objects from the ShapeNet dataset 7 as deformation targets and then apply our cage deformer to edit the implicit geometry stored in the canonical radiance field. As shown Figure [ our framework capable of editing the implicit geometry of object in the canonical space by deforming the radiance field learned by our neural renderer module. The deformed results resemble the selected target aspects size, shape, and scale. These results demonstrate that our framework can edit the radiance field such that the deformed geometry resembles the target.
4.2 Neural Animation
Synthesized Object Animation. To quantitatively determine the animation precision, we analyze the rendering quality of an animated sequence and compare it to the traditional linear blend skinning (LBS) animation method which operates on the explicit deformable mesh. We train our neural renderer using multiview images of the object rendered at the starting frame. To animate the object, we extract the animated mesh from the animation sequence and use it as the deformed geometry g' Our cage deformer then builds the MVC field using Equation 10 We animate our object by applying the extracted meshes g' sequentially our framework.
As is shown in Figure our framework capable of deforming the radiance field and generating continuous animation of objects. CageNeRF also achieves nice render quality which is comparable to that driven by explicit animation methods. As shown Table I our framework achieves promising results in animating the neural radiance field when compared to explicit mesh deforming methods.
Human Animation. We also evaluate on real-world dataset (i.e., Human 3.6M). and compare with the state-of-the-ee method AniNeRF [32] as well as other human sythesis methods [4] 49]. The Human 3.6M dataset is captured by synchronized cameras and has 2D and 3D joint annotations. We first estimate the geometry proxy of the posing human using the SMPL model by optimizing the 3D joint error and 2D re- re-projection joint error. Then, we utilize the geometry proxy the "novel pose" and apply the operation in Figure to generate deformed cage with the matching pose. Then we build the MVC field using this deformed cage and posing geometry proxy. Since our model only stores the information inside the canonical space, CageNeRF can generalize to novel poses without additional training. This is in contrast to AniNeRF, which needs to optimize an additional blend weight network using the estimated geometry in order to synthesize novel pose results.

---

Deformation Source
Reenactment
Targets
Deformation
Results
Figure 9: Qualitative results of animation reenactment using SMPL models as the reenactment target. Top Row shows SMPL models sampled from Surreal dataset as the reenactment target.
Bottom Row reenactment results defromed from the souce model shows on the right.
Table 2: Quantitative performance comparison on Human 3.6M dataset. "NT", "NHR" and "AniNeRF" refer to Neural Textures, Neural Human Rendering and Animatable NeRF respectively.
PSNR
SSIM
NT 41
NHR [49]
AniNeRF 32
Ours
NT
NHR
AniNeRF
Ours
S5
19.87
20.64
23.27
23.01
0.855
0.872
0.892
0.877
S7
20.47
20.29
22.50
22.07
0.856
0.868
0.890
0.873
S9
22.96
23.04
24.72
23.58
0.873
0.879
0.908
0.886
S11
21.71
21.91
24.55
24.19
0.859
0.871
0.902
0.891
Avg. 21.25
21.47
23.76
23.21
0.861
0.873
0.898
0.882
As shown in Figure and Table 2 the performance of our framework in synthesizing dynamic human body is comparable to AniNeRR across different subjects. CageNeRF achieves slightly lower PSNR due the inaccurate estimation of the geometry used in our cage deformer module. Moreover, we employ SMPL model as the geometry proxy during cage deformation, where SMPL only models human body, while the performer in Human 3.6M dataset wears loosely fitted clothes which has complex nonrigid deformations during motion. The inaccuracy in the geometry proxy further induces sampling displacement during volumetric rendering and thus impacts the performance of our framework. Despite the fact that our framework only takes coarse estimation of the object, it still able correctly deform and synthesize complex dynamic objects such as the human body.
4.3 Pose Reenactment
To analyze the generalization ability of CageNeRF, we apply our framework to tasks of pose reenact- ment and motion retargeting from humans to humanoid objects. We keep the same training process as Section 4.1 Afterward, we randomly sample posed SMPL models from the Surreal dataset as the target for deformation transfer. Both the posed SMPL model and the optimized robot cage are fed into our cage deformer to calculate the cage deformation. Finally, the barycentric interpolation applied deform the static robot model using Equation g As is shown in Figure 9 although the structure of the source robot is notably different from the target human, the deformed pose and geometric structure correspond well to the given target, while keeping the source identity and texture.
4.4 Impact of the MVC Sampling Strategy
To evaluate the effectiveness of our anti-aliasing MVC sampling strategy, we generate an MVC field using different sampling strategies and qualitatively and quantitatively compare the rendering results. As shown in Figure IO] the rendering result generated with grid-sampled MVC field displays inflated geometry and blurred texture, due to the inaccurate mapping between the canonical and the deformed space. In contrast, the result rendered using our anti-aliasing sampled MVC field has notably sharper textures and more accurate geometry structure. also shown in Table 3 the anti anti-aliasing sampling strategy significantly increases the render quality all three evaluation metrics.

---

Ground Truth
Anti-aliasing Sampling
Grid Sampling
Figure 10: Comparison between Anti-aliasing Sampling and Grid Sampling, Our anti anti-aliasing sampling strategy achieves notably better rendering results w.r.t. geometry and texture.
Table 3: Quantitative results of animated hulk using different MVC sampling strategies. AA Sampling: the anti-aliasing sampling strategy.
Metrics
Grid Sampling
AA Sampling
PSNR
30.695
35.280
SSIM
0.916
0.956
LPIPS
0.0293
0.0156
Limitations
Our approach achieves high quality results in deforming and animating implicit objects by combining cage-based deformation with neural radiance field. However, there remains a few limitations of our approach we would like to address. 1) While the cage-based deformation is category-agnostt and easy to generalize under various settings, it is not suited for modeling complex and subtle deformations such as facial expressions. Our work is proof of the concept that explicit deformation can be applied to implicit neural representations. We believe our framework is also compatible with other explicit deformation schemes, which may provide the extra capability in adapting complex movements. 2) Since the color and density of the canonical radiance field are relatively stable, our framework is not suited for modeling dynamic objects with changing color rendering objects under varying lighting conditions. Recent works [39, (55) have developed novel techniques on factorizing basic color and density information into several decoupling elements such as surface normal, albedo, and light visibility, this further enables material editing and neural scene relighting. It would be interesting further improve and augment our framework have editable material, color, and lighting by incorporating novel neural rendering algorithms and structures.
Conclusion
We propose versatile framework for deforming and animating implicit field. Comparing previous methods, our framework excels in aspects of generalization, structural flexibility, and editing capabil- ity. This is achieved by adopting a deformable cage the universal constraint for our radiance field editing which does not require any data-depente deteedeeedd prior knowledge. We demonstrate the effectiveness of our framework in the tasks of object animation, geometry editing, and deformation transfer. Ex- perimental results show that our framework is capable of animating and deforming arbitrary objects represented by implicit field, and produces results comparable other state-of-the-art methods.
Acknowledgments and Disclosure of Funding
This work is supported by NSFC (62225112, 62101325, 61831015, 62201342, U19B2035), the National Key Research and Development Program of China (2021 YFE0206700), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102) and CCr-Alibaba Innovative Research Fund For Young Scholars. Authors would like to appreciate the Student Innovation Center of SJTU for the GPU support.
10

---

References
[1] ShahRukh Athar, Zhixin Shu, and Dimitris Samaras. Flame-in-nerf: Neural control of radiance fields for free view face animation. CoRR, abs/2108.04913, 2021.
[2] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, pages 9835-5844, 2021.
[3] Matthew Berger, Andrea Tagliasacchi, Lee M. Seversky, Pierre Alliez, Gaël Guennebaud, Joshua A. Levine, Andrei Sharf, and Cláudio T. Silva. survey of surface reconstruction from point clouds. Comput. Graph. Forum, 36(1):301-329, 2017.
[4] Volker Blanz and Thomas Vetter. SIGGRAPHI pages 187-194, 1999.
morphable model for the synthesis of 3d faces. In
[5] Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. FAUST: dataset and evaluation for 3d mesh registration. In CVPR, pages 3794-3801, 2014.
[6] Mario Botsch, Leif Kobbelt, Mark Pauly, Pierre Alliez, and Bruno Lévy. Polygon Mesh Processing. 2010.
[7] Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese. Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. CoRR, abs/1512.03012, 2015.
[8] Xu Chen, Yufeng Zheng. Michael J. Black, Otmar Hilliges, and Andreas Geiger. SNARE: differentiable forward skinning for animating non-rigid neural implicit shapes. In ICCV, pages 15574-15544 2021.
[9] Lin Gao, Yu-Kun Lai, Dun Liang Shu-Yo Yu Chen, and Shihong Xia. Efficient and flexible deformation representation for data-driven surface modeling. ACM TOG, 1515):158:11118:11, 2016.
[10] Lin Gao, Yu-Kun Lai, Jie Yang. Ling-Xiao Zhang, Shihong Xia, and Leif Kobbelt. Sparse data driven mesh deformation. IEEE TVCG, 27(3):2055-2100, 2021.
[11] Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. 3d-coded: 3d correspondences by deep deformation. In ECCV, volume I 1206, pages 235-251, 2018.
[12] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In ICCV, pages 5784-579 2021.
[13] Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: A survey. IEEE TPAMI, 43(1))43338 4364, 2021.
[14] Yang Hong. Bo Peng. Haiyao Xiao. Ligang Liu, and Juyong Zhang. Headnerf: real-time berf-based parametric head model. CoRR, abs/21 1ass/212..0567,. 2021.
[15] Kurt Hornik, Maxwell B. Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Newral Networks, ((5):359-366 1989.
[16] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchiseseu. Human3 3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE 36(7:
2014.
[17] Alec Jacobson, Ilya Baran, Ladislav Kavan, Jovan Popovic, and Olga Sorkine. Fast automatic skinning transformations. ACM TOG, 31(4):77:1 2012.
[18] Wonhong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In ICCV, pages 12929-12938, 2021.

---

[19] Timothy Jeruzalski, David I. W. Levin, Alec Jacobson, Paul Lalonde, Mohammad Norouzi, and Andrea Tagliasacchi. Nilbs: Neural inverse linear blend skinning. CoRR. abs/2004.05980, 2020.
[20] Pushkar Joshi, Mark Meyer, Tony DeRose, Brian Green, and Tom Sanocki. Harmonic coordi- nates for character articulation. ACM TOG, 26(3):71, 2007.
[21] Tao Ju, Scott Schaefer, and Joe D. Warren. Mean value coordinates for closed triangular meshes. ACM TOG, 2463):561-566, 2005.
[22] Ladislav Kavan, Steven Collins, Jirí Zára, and Carol O'Sullivan. Geometric skinning with approximate dual quaternion blending. ACM TOG, 17(4):15:11-105222 2008.
[23] Ladislav Kavan, Steven Collins, Jiff Zára, and Carol O'Sullivan. Geometric skinning with approximate dual quaternion blending. ACM TOG, 27(4):1-23, 2008.
[24] Tianye Li, Timo Bolkart, Michael J. Black, Hao Li, and Javier Romero. Learning model of facial shape and expression from 4d scans. ACM TOG, 3666):194:11194:11,,, 2017.
[25] Yaron Lipman, David Levin, and Daniel Cohen-Or. Green coordinates. ACM TOG, 27(3):78, 2008.
[26] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: skinned perti- linear model. ACM TOG, (0:::::2:::112::1111 2015.
[27] William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. In SIGGRAPHI pages 163-169, 1987.
[28] Bruce Merry, Patrick C. Marais, and James E. Gain. Animation space: A truly linear framework for character animation. ACM TOG, 25(4))1400-1425 2006.
[29] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, pages 405-421, 2020.
[30] Ramakrishnan Mukundan. 3D Mesh Processing and Character Animation With Examples Using OpenGL, OpenMesh and Assimp. 2022.
[31] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Steven M. Seitz, and Ricardo -arail- Nerfies: Deformable neural radiance fields. In ICCV, pages 5845-5854, 2021.
[32] Sida Peng. Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodics. ICCV, pages
2021.
[33] Sida Peng, Shangzhan Zhang, Zhen Xu, Chen Geng, Boyi Jiang, Hujun Bao, and Xiaowei Zhou. Animatable neural implicit surfaces for creating avatars from videos. CoRR, abs/2203.00 2030000.81.. 08133, 2022.
[34] Sida Peng. Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, pages 9054-9063. 2021.
[35] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in metric space. In NewrlPS, pages 5099-5108, 2017.
[36] Shunsuke Saito. Jinlong Yang, Qianli Ma, and Michael Black. Scanimate: Weakly supervised learning of skinned clothed avatar networks. In CVPR, pages 2886-2897, 2021.
[37] Franco Scarselli and Ah Chung Tsoi. Universal approximation using feedforward neural networks: survey of some existing methods, and some new results. Neural Networks, 11(1):15-37, 1998.
12

---

[38] Daeyun Shin, Charless C Fowlkes, and Derek Hoiem. Pixels, voxels, and views: A study of shape representations for single view 3d object shape prediction. In CVPR, pages 3061-3069, 2018.
[39] Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, pages 7495-7500 2021.
[40] Jean-Marc Thiery, Julien Tierny, and Tamy Boubekeur, Jacobians and hessians of mean value coordinates for closed triangular meshes. CoRR, abs/1 109.1019.1191 1914, 2011.
[41] Justus Thies, Michael Zollhöfer, and Matthias NieBner. Deferred neural rendering: image synthesis using neural textures. ACM TOG, 38(4))66:1 2019.
[42] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In ICCV, pages 12959-12970, 2021.
[43] Wei-Cheng Tseng, Hung-Ju Liao, Lin Yen-Chen, and Min Sun. Cla-nerf: Category-level articulated neural radiance field. CoRR, abs/2202.00181, 2022.
[44] Güil Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In CVPR, pages 4627-4635, 2017.
[45] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NIPS, pages 27171-27181 27183, 2021.
[46] Xiaohuan Corina Wang and Cary B. Phillips. Multi-weight enveloping: least- squares approxi- mation techniques for skin animation. In SIGGRAPH, pages 129-138, 2002.
[47] Yifan Wang, Noam Aigerman, Vladimir G. Kim, Siddhartha Chaudhuri, and Olga Sorkine- Hornung. Neural cages for derail-pr 3d deformations. In CVPR, pages 72-80, 2020.
[48] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4):600-612, 2004.
[49] Minye Wu, Yuehao Wang, Qiang Hu, and Jingyi Yu. Multi-view neural human rendering. In CVPR, pages 1679-1688, 2020.
[50] Kai Xu, Vladimir G. Kim, Qixing Huang, Niloy J. Mitra, and Evangelos Kalogerakis. Data- driven shape analysis and processing. In SIGGRAPH Asia, pages 4:1-4:38, 2016.
[51] Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao. Facescape: A large-scale high quality 3d face dataset and detailed riggable 3d face prediction. In CVPR, pages 598-607, 2020.
[52] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. NeurIPS, 34, 2021.
[53] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. CoRR, abs/2010.07492, 2020.
[54] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The un- reasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586-595, 2018.
[55] Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul E. Dehevec, William T. Freeman, and Jonathan T. Barron. Nerfactor: neural factorization of shape and reflectance under an unknown illumination. ACM TOG, 40(6):23::1 2021.
[56] Yuzhe Zhang, Jianmin Zheng, and Yiyu Cai. Proxy-driven free-form deformation by topology- adjustable control lattice. Comput. Graph.. 89:167-177. 2020.
[57] Yufeng Zheng, Victoria Fernández. Abrevaya, Xu Chen, Marcel C Bühler, Michael J Black, and Otmar Hilliges. IM avatar: Implicit morphable head avatars from videos. CoRR, 12777,,,07771,, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes], our work focuses on providing editing method for implicit 3D representations in a generalized manner, it is possible that someone utilizes our method to produce fake videos.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We do not include theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We describe the implementation details in the supplementary material. The data utilized is publicly available, and we will make
code publicly available.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] We discuss the training details in supplementary material.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? [Yes] See supplementary for detailed descriptions.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We use existing datasets and cited them in our paper.
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We discuss this in the supplementary material.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] The data used in our work does not contain such information.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [No] Our work does not involve crowdsourcing as such, same follows.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [No]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [No]
14


Reviews:
[{"review_id": "xQdx6IzOvfY", "reviewer": "Reviewer_krXf", "summary": "The paper proposes a way to deform Neural Radiance Fields using a cage-based paradigm. It is based on the training of a cage deformer, optimized on the poses. Then, given a new pose, the method regresses the cage on an explicit geometry and computes a dense mean-value coordinate field to compute the new visualization. The method is tested on articulated (i.e., humanoids) and non-articulated (i.e., chairs) objects.", "questions": "- Why do some of the results present hole artefacts?\n- Why the method is not tested on more complex movements? There is any limitation to taking them into account?", "limitations": "The paper fairly discusses limitations and societal impact, while on the first probably a bit more elaboration on the kind of animations admitted by the method can be discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "== Strenghts ==\n\n- The approach seems novel and addresses a complex problem. Providing novel tools to edit Neural Radiance Fields is a hot topic with a potentially great interest for the computer graphics community\n- The method seems applicable in different contexts, with good results both for rigid objects and non-rigid objects\n- The method is well discussed in its elements, providing ablation and justifying the design choices.\n\n== Weaknesses ==\n- Looking at the video, there seem to be some hole artefacts, e.g., under the lifted arm of Hulk, in the pose reenactment experiment. Is it due to the stretching introduced by the paradigm, or is it because the NERF itself does not well represent some areas?\n- The method still requires access to an explicit geometry, which somehow limits the arbitrary resolution power typical of neural representations. When an SDF is used, it requires a second training for the representation. \n- The shown animations are not particularly complicated; probably, the use of cages requires to limit of the intersections, but I cannot understand if the method is applicable also in more complex scenarios with more articulated movements.\n\n\n== minor fixes ==\n- Line 174: missing a comma after \"$o$ is the camera origin\"\n- Line 291: typo, \"Impact\"\n\n== After Rebuttal ==\n\nAfter the rebuttal and the discussion phase, I am prone to keep my initial rating and vote for acceptance. For the final version, I suggest including in the main manuscript the observations discussed in the reviewers-authors discussion, especially about the limitations and trade-offs of the method. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fs7tUxaUoZ", "reviewer": "Reviewer_gcEA", "summary": "CageNeRF allows for deforming a neural radiance field of arbitrary surfaces/objects, unlike prior work that was restricted to specific categories like humans or faces by design. The deformation network is trained per object and requires training data for deformations. The method design uses an invertible/bi-directional deformation parametrization based on classic graphics knowledge, namely deformation cages. It is an extension of the prior work Yifan et al. (Neural Cages for Detail-Preserving 3D Deformations; CVPR'20) to implicit fields. This extension is non-trivial due to the volumetric and non-discretized nature of the neural radiance field. The method is evaluated on deformation transfer, reenactment, and animation. All these applications apply a deformation cage to a neural radiance field, with the deformations coming from the same sequence as the radiance field in the case of synthetic reenactment and from a different sequence/scene in all other cases.", "questions": "* An argument why a comparison against face-specific NeRFs is not necessary would be good.\n* What is the speed of the method? By what factor does the rendering slow down due to the deformations, i.e. relative to just directly using the canonical NeRF, skipping C in Eq. 15?\n* I don't understand how the grid sampling (L195) interacts with the volume rendering. The points p from the rendering are first along a straight ray. How are these then transformed via the cage into the canonical radiance field? The points p won't lie on a grid but anywhere in space, so how does the grid come into play? I understand the thresholding with epsilon.\n\nMinor questions that are relevant for a revision but not the rebuttal:\n* Eq. 5 is not easy to parse. \"v\" is used for the vertices of the cage template, the proxy geometry, and as any individual vertex of the proxy geometry. I cannot tell which of these \"v\" the \"max\" is over. I believe it should be the vertices of the proxy geometry.\n* I also don't understand how this is turned into a neural-network optimization problem. What is the loss? If this information can be found in Neural Cage, I strongly recommend to add a Background section explaining the important parts of Neural Cage that are used by this submission.\n* How is the cage deformer applied in Sec. 3.2 (L137f)? Is C_t the \"Optimized Cage\", G_p the \"Novel Pose\", and C_can the \"Deformed Cage\"? If so, this is not at all obvious and needs to be stated explicitly.\n* In Sec. 3.3, what is the loss, during pretraining and for Sec. 3.2? \n* In Fig. 3, both the lower and upper branch use E_C to my understanding. It would be helpful to the reader to visualize both in the same way, e.g. use the \"E_C\" block also for the upper branch instead of the words \"Feature Extraction\".\n* In Sec. 3.3, please mention that a global PointNet++ encoder is used. It was not clear to me for a while whether the deformation might be regressed per-vertex (coordinate-based MLP) instead for all vertices at once. The supplement also gives the unclear description \"to predict the vertex-wise offset for the input cage base\" (L85). \n* In L216ff, G_d, G_p and G_s are used incorrectly, I believe. G_d should be G_n at the very least.\n* Are the numbers in Table 1 from rendering into a single view or multiple views per frame index?\n\nTypos etc.:\n* L170: Not a full sentence.\n* L291: Impact", "limitations": "Limitations are mentioned but I believe that it should be also explicitly mentioned that the cage deformer might not generalize well to target poses outside the training distribution since there is no evidence to the contrary in the submission.\n\nGeneral broader impact is discussed but no potential negative societal impacts are mentioned. Deformation transfer could have the typical deep fakes issue where the identity of one person is animated by motions of another.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Editing the neural radiance field of arbitrary surfaces/objects is an important direction for the NeRF field.\n* CageNeRF is a fairly elegant parametrization and tackles e.g. aliasing issues well. \n* Only human reenactment allows for a comparison against prior work, namely AniNeRF for human-specific NeRF animation. CageNeRF performs worse but close enough in my opinion, considering that it is a general method.\n* The paper is written in good English.\n\nWeaknesses:\n* The method section skips over non-obvious steps in many places, see below in Questions.\n* Deformation transfer is demonstrated only qualitatively and only on the ShapeNet chair category for a handful of (random) instances. In addition, this leads to deformations that seem almost just like per-axis scalings. Other categories with more interesting deformations would show that what the method does goes beyond what an anisotropic matching of the object bounding boxes could do. I don't believe the deformation transfer results are sufficient evidence that deformation transfer actually works with the method.\n* The quantitative results in Table 1 are on 3-4 specific frame indices out of dozens of indices per sequence (on top of evaluating only two sequences). That leaves the door wide open for cherry-picking a few good indices. Averages (and standard deviations) across the entire sequences should be reported. \n* The deformed cage is determined by a neural network and hence might not generalize well to deformed target poses that lie far away from the training data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8Q9Oavka_v2", "reviewer": "Reviewer_1oeX", "summary": "The paper extends the NeuralCages work to NeRF to enable the deformation of implicit fields. Given a NeRF, it first extracts surface points and then optimizes a template cage to enclose the points by penalizing the negative MVC values of the sampled points. It then follows NeuralCages to deform the cage. All points in the cage can be deformed accordingly with barycentric coordinates and the vertex position of the deformed cage. The author proposes a sampling trick to handle the aliasing issue (I don't understand this part).\n\nIn general, the idea of combing NeRF and NeuralCages is straightforward. However, the presentation is far from satisfactory. The experimental results are not convincing or impressive.", "questions": "See weaknesses. ", "limitations": "The authors have mentioned some limitations in supplementary materials.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe idea of introducing cage deformation to NeRF is interesting. The overall idea of combing NeRF and NeuralCages is straightforward and technically sound.\n\nWeaknesses:\n1. Although the overall idea is simple, the presentation is far from satisfactory and hard to follow. Especially, the terms and symbol system are complicated and confusing:\n\n(a) What do the terms `geometry` and `geometry proxy` refer to? Are they refer to surface meshes only, surface points, or the regions/points within surface meshes?\n\n(b) geometry \\mathcal{G}_{can} is a set of surface points? but the geometry proxy \\mathcal{G}_{p}(\\mathbf{v}, \\mathbf{f})  is a mesh?  What about deformed geometry \\mathcal{G}_{n}? Again, the meaning of symbel \\mathcal{G} is confusing. \n\n(c) If you want to say a mesh M with vertices V and faces F, please use M={V, F}. M(V, F) seems like a function taking two arguments. For example, the text uses \\mathcal{G}_{p}(\\mathbf{v}, \\mathbf{f}), which is strange.\n\n(d) Line 132: which nearest neighbor algorithm? Please add a reference. Does it generate a watertight manifold mesh?\n\n(e) Line 139: sample points from mesh surfaces or the interior. If sample points from the interior, why \\mathcal{G}_{p} is closed mesh? If sample points from surfaces, can we directly sample from \\mathcal{G}_{can}? And there is no need to construct \\mathcal{G}_{p}?\n\n(f) Equation 5: does \\mathbf{v} refer to a sampled point? If so, please use another symbol instead of \\mathbf{v}, which refers to vertices before. Also, I cannot understand why it is ``max min``? We need to penalize all negative MVC values, and it should be ``min min`` instead?\n\n(g) Line 157: what is \\mathbf{E}_{C}?\n\n(h) For the deformed geometry \\mathcal{G}_{n}, what does `n' refer to? It's kind of confusing.\n\n(i) Line 219-220 says \\mathcal{G}_{s} and \\mathcal{G}_{p}. But in the caption of Figure 4, it says \\mathcal{G}_{s} and \\mathcal{G}_{n}. Which one is correct?\n\n(j) Equation (14) what does `g' in \\mathbf{p}_{n}^{g} refer to?\n\n(k) Do we really need \\mathcal{C}(\\cdot)? Only mentioned once.\n\nIn general, section 3.2 - 3.4 is very confusing and wordy. I tried hard to follow but still failed to follow all the contents. I suggest the authors rewrite it more precisely, clearly, and compactly. A figure to illustrate the relations between the symbols may also help.\n\n2. In section 3.3, the network architecture, loss functions, and training details are missing.\n\n3. I can roughly understand the reason for the aliasing issue. But why do we need a vicinity threshold \\epsilon? If \\epsilon is very small, do we still have the aliasing issue? Moreover, I have no idea of how and why it is addressed by introducing a shell \\mathcal{G}_{s}. Motivations and discussions are missing.\n\n4. The setting of ``Synthesized Object Animation'' is kind of confusing. If we already have a sequence of animated meshes, we can directly render the images. Why do we still need the proposed CageNeRF method? I am also confused about how the method actually works (Line 256-258).\n\n5. For Pose Reenactment, do we have meshes of humanoid objects? If so, can we directly apply NeuralCages? Why do we need CageNeRF?\n\n6. Both Table 1 and Table 3 are computed over a single shape? What about Table 2? The experiment results are not convincing or impressive.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept