Template-free Articulated Neural Point Clouds for Reposable View Synthesis
Authors: Lukas Uzolas
Elmar Eisemann
Petr Kellnhofer
Venue: NeurIPS 2023 poster (2023)
Keywords: Radiance Fields
View Synthesis
Kinematics
Reposing
NeRF
TL;DR: A method for fast learning of class-agnostic articulated models for high-quality reposing and novel view synthesis of captured 3D objects.

Abstract: Dynamic Neural Radiance Fields (NeRFs) achieve remarkable visual quality when synthesizing novel views of time-evolving 3D scenes. However, the common reliance on backward deformation fields makes reanimation of the captured object poses challenging. Moreover, the state of the art dynamic models are often limited by low visual fidelity, long reconstruction time or specificity to narrow application domains.  In this paper, we present a novel method utilizing a point-based representation and Linear Blend Skinning (LBS) to jointly learn a Dynamic NeRF and an associated skeletal model from even sparse multi-view video. Our forward-warping approach achieves state-of-the-art visual fidelity when synthesizing novel views and poses while significantly reducing the necessary learning time when compared to existing work. We demonstrate the versatility of our representation on a variety of articulated objects from common datasets and obtain reposable 3D reconstructions without the need of object-specific skeletal templates.

Template-free Articulated Neural Point Clouds for Reposable View Synthesis
Lukas Uzolas
Elmar Eisemann Delft University of Technology The Netherlands
Petr Kellnhofer
(l.uzolas, e.eisemann, p p @tddelllle@@@eeellee @tudelff wt..udllf......
Abstract
Dynamic Neural Radiance Fields (NeRFs) achieve remarkable visual quality when synthesizing novel views of time-evolving 3D scenes. However, the common reliance on backward deformation fields makes reanimation the captured object poses challenging. Moreover, the state of the dynamic models are often limited by low visual fidelity, long reconstruction time or specificity to narrow application domains. In this paper, we present a novel method utilizing point-based represen- tation and Linear Blend Skinning (LBS) to jointly learn Dynamic NeRF and an associated skeletal model from even sparse multi-view video. Our forward -wrrpnng approach achieves ffae-oof-the-ar visual fidelity when synthesizing novel views and poses while significantly reducing the necessary learning time when compared existing work. We demonstrate the versatility of our representation on a variety of articulated objects from common datasets and obtain reposable 3D reconstructions without the need of object-specific skeletal templates. The project website can be found https:::// /tt////lkas../// uzolas www
-eittteddd-----------o MeinttNeet -NeRF/
Introduction
Synthesizing novel photo-realistic views captured 3D scenes important for many domains includ- ing virtual/augmented reality, video games movie productions. In recent years, Neural Radiance Fields (NeRFs) [ have proved their remarkable capacity to represent complex view-dependent effects, captured in photographs and videos, sparsely sampled from natural light fields [2]. Follow-up works have extended the scope to dynamic scenes [O], facilitating rendering of unseen views at different timestamps. Despite progress in reconstruction quality and speed [11], manipulating learned scenes remains a challenge, but would be a highly desirable feature, as it can enable downstream applications, such the creation of avatars for virtual presence 3D assets for games and movies.
The key challenge of reposing a NeRF inverting the backward- warping function that maps individual observations to shared canonical representation 3 It is inversion of traditional kinematic animation, such as Linear Blend Skinning (LBS) [2], where canonical shape is wwward-warped to desired pose. Such inverse mapping often requires resolving ambiguous situations it is difficult to guarantee bijectivity (see Fig. 2). A common remedy are parametric templates, typically built for narrow application domains, such human heads and bodies [3) but they are difficult to generalize. Alternatively, object shapes can retrieved from videos as ensembles geometric parts, yet existing techniques provide limited amage-synthesis fidelity 117.99. Our work aims combine these different lines work and enable joint learning of the NeRF representation and its pose parameterization from sparse dense multi-view videos. We aim for time-efficient class-agnostic view synthesis of reposable models with high image-synthesis quality without access to template or pose annotation, which is combination not currently covered by existing work (see Table
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Point Cloud Extraction
b) Kinematic Neural Point Cloud Training
R
Pose Regressor
7
Color and Density Regressor
Kinematic Warp
Dense Sampling
NeRF
Feature Point Cloud
Learned Skinning
Extracted Skeleton
Weights
Rendered Image Weights
Overview of our method: a) First, we pre-train NeRF backbone initialize feature point cloud pe for selected canonical timestamp and to extract an initial skeleton. b) During the main training stage, Pe forward- -arred using LBS consisting of learned lime-invariant skinning weights tû; and time-dependent pose transformations from an MLP regressor 4,. The image is obtained by integration and decoding of features aggregated from points along each camera ray. In summary, we fine-tune the initial neural point features f, skinning weights w;, joints J, density and color regressor & and & of the backbone. We fully train the pose regressor 4, and feature point decoder dp from scratch. In test time, we modify the pose transformations obtain novel poses.
Figure 1:
Method
Pose Generic Fildelity Training
LASR 17
No
Yes
Shape
<
VISER [8
No
Yes
Shape
2-12
BANMO 22
No
Yes
Low
2 12
D-NeRF 3
No
Yes
High
2 12
TiNeu Vox 11
No
Yes
High
2
Observation
Observation space
Canonical space
CASA 23
Yes
No
Shape
2
space
space
LASSIE 19
Yes
Yes
Low
2 12
Figure 2: Examples of poses difficult for back- ward warping. (a) Ill-defined projection from an observation to the canonical space. Both am- biguous solutions (green and magenta) correctly pass through the semantically corresponding surface points and C. (b) Projection ambigu- ity for points of contact between two surfaces. Note that in contrast, it is trivial to forward warp the object points from a well-chosen canonical space any observation space.
WIM 20
Yes
Yes
High
122h
Ours
Yes
Yes
High
22h
Table I: comparison of representative dy-
namic 3D representations learned from 2D videos. We analyze reposibility, agnosticism to object class, image synthesis fidelity and train- ing time. Papers that demonstrate reposing are shown below the bar. "Shape" denotes shape- only reconstruction without texture details.
Recent work by Noguchi et 20 addresses similar problem. However, we exploit the structure- free nature of point-based NeRF representations (21), enabling direct forward warping of the canonical object to any desired pose, while maintaining the capacity and ficxibility of NcRF-like rendering to capture highly detailed image features. Together with automatically extracted and jointly- optimized LBS skeletal pose parameterization. our work allows for faster training and achieves state-of-the-art visual quality in a much shorter time. Finally, demonstrate how easily repose learned representations of varied objects by directly editing joint angles of the forward LBS model.
To summarize, we make the following contributions: 1) We propose novel method for learning articulated neural representations of dynamic objects using forward projection of neural point clouds. 2) We demonstrate state-of-the-r of-the-art novel view synthesis for variety of multi-view videos with training times lower than comparable methods. 3) We jointly learn a skeletal model without domain- specific priors annotations and demonstrate reposing of the reconstructed 3D objects.

---

2
Related Work
Neural Radiance Fields and Parameterizations Implicit neural networks have recently emerged a powerful tool for building differentiable representations of 3D scenes 47 and for learning view-dependent Neural Radiance Fields (NeRFs) 2 48|57] enabling photo-realistic novel view synthesis. Their high computational cost has been dramatically reduced by spatial decomposition [58]- 60] and hybrid representations [61-655 Recently neural point clouds have combined expressive neural functions with the flexibility of explicit geometry [1 66)69) We also use neural point clouds, but unlike previous work, we jointly learn a forward kinematic model along with the associated soft part segmentation and, thus, convert fast voxel-based representations to reposable NeRFs
Dynamic Neural Radiance Fields Equivalent to 2D video, dynamic NeRFs capture temporal changes in scenes by encoding each frame separately [70), expanding the radiance field into the temporal domain I0]. Alternatively, single canonical NeRF can be animated by backward warping from the canonical space to individual time-varying observation spaces 6 on the bijectivity of the mapping which is difficult to guarantee for all observed poses (see Fig. 2 Forward mapping only requires single well well-posed canonical representation, which we exploit.
72,, or time-dependent interpolation in a compact latent representation 6
However, such warps rely
Object Reposing Directly reposing nigh-dimensional deformation fields of dynamic NeRFs is impractical. Instead, parametric templates can sparsely represent prominent deformation modes for faces 13 14], bodies [6], hands [73], and also non-human objects, such as animals [74). Together with skeleton-based LBS, they enable articulated neural representations of human heads [69 75481
or bodies (2-777) as well as modeling distributions generative models [98 However, the assumption of piece-wise rigid motion and the availability of a skeletal template restrict the applicable object classes. The former issue has been addressed by physically inspired but computationally expensive deformation models [102]. To remedy the latter issue, the skeletal model can be retrieved from a database [23], adapted from a generic graph [19] [03], fitted as a template to 2D observa- tions [104 [08] fully learned during 3D shape recovery (20 [09]. An external cage can also be used for re-animation of static NeRFs [110]. Alternatively, a surface embedding can be learned
without skeleton if reposing is not required [17 [8 22).
Similarly to Noguchi et al. (2), we jointly learn a 3D object representation, a skeletal model, and observed poses. However, we utilize point-based NeRF representation to benefit from its computational efficiency, ease of geometric transformation, and capacity to learn complex light fields.
We share our point- approach with a contemporaneous work MovingParts (II1) which, however, relies on inverting the backward flow. We demonstrate the reconstruction of large transformations in the Robots dataset [20). not demonstrated in MovingParts. Furthermore, our approach does not enforce binary segmentation and, hence, can represent non-rigid part transitions (see Fig. right).
3 Preliminaries
Our method builds upon available NeRF reconstruction methods as a backbone for learning a reposable dynamic representation. In principle. any NeRF-like representation a suitable initial point for our training. In practice, we use TiNeu Vox [IT] in all experiments, as it enables fast reconstruction with both sparse and dense multi-view supervision in dynamic scenes. Next, we describe the core concepts of NeRF 1 and TiNeu Vox [1] to provide context for our method in Sec. 4
Time-Aware Neural Voxels NeRF 1 maps point (z, y, =) and view direction d (0, Φ) to color (r,g, g, b) and density a. The mapping function often takes form of a Culti-Layer-Perceptree (MLP) t. such that (c, (pp, d). The color of an image pixel (rr) can be obtained via volume rendering along the ray r(t) = o td.
TiNeu Vox [ expands NeRF to dynamic scenes and significantly increases training speed by utilizing an explicit volume representation. To this extent, each point p at time in observation space is wookward-wappee to canonical space as
nd = T (p. t), where P = 7p). t- (t)(
(1)
3

---

Here, is the positional encoding M. P, small color-decoding MLP, Ds a backward warping MLP, and p and the encoded point and time respectively. The canonical point p is used to sample feature vector Vm R from the explicit feature volume V xxXXXxX X2XV by means of trilinear interpolation interp with varying stride Sm. at each scale: Vm = (interrp(p" V sm)). The feature vectors across all scales are concatenated v V1 θ v ... @ Vas, where + is the concatenation operator. Finally, the view-dependent color and density are regressed by additional MLPs y, & d and &e:
f (v, t). where o = Pf(f). c df.
(2)
The volume rendering equation [12] integrates density/color of points p. sampled along the ray r:
N
(rr)
T:(1 - (a0-erp(-oo;)))) where T; = eapp Σ Σ
o(,ر)), j=1
Here, òe is the distance between P. and pi+1- The feature volume V and all MLPs are optimized end-to-end and supervised by the Mean Squared Error (MSE) and the corresponding ground-truth pixel color C(r): Cphoto ||C(r) C(r)
Note that because & (Eq. conditioned by the time i, space and time are entangled and the feature volume V does not represent& single static canonical shape. Therefore, we cannot directly learn forward warping function invert 4 as proposed by Chen et al. 1024 In the next section, we show that this is not necessary and we can learn a forward kinematic model directly without relying on specific properties of the backbone, which will allow to replace it in the future.
4 Method
While dynamic NeRF models such as ToNeuVox ||| can reproduce motion a dynamic scene, they do not enable reposing. Furthermore, it is impractical to post hoc reparametrize their motion representation due to the ambiguities of backward flow inversion (see Fig. 2).
We instead propose completely new representation based on a neural point cloud and an explicit kinematic model for forward warping, and we use the backbone only as an initialization for our training procedure (see Fig. T]. First, we extract feature point cloud from selected canonical frame of the backbone Sec. [4.]] Second, we describe the underlying kinematic model of our 3D representation and its initialization Sec. 4.2 Consecutively, introduce how the 3D point cloud is warped from a canonical space to an observation space and rendered Sec. 4.3 Lastly, we describe our losses Sec. 44
4.1 Canonical Feature Point Cloud
We first pre-train backbone NeRF model Vox II] using its default parameters. To initialize the feature point cloud, we sample the canonical density function of Tixeu a = (f) (Eq. ( on uniform coordinate grid and discard empty samples through thresholding. The grid resolution is adaptively chosen, such that PP| x 10000, similarly to other explicit NeRFs [1]] 64 [13]. After
thresholding, obtain points P {p:li 1,.., Nj. Furthermore, we extract feature vector f; for each point (see Eq. 2
4.2 Kinematic Model
We forward- warp our feature point cloud from canonical space to observation space using an LBB kinematic model to match the training images. Here, we describe how we initialize, use, and simplify our kinematic model.
Skeleton Initialization We do not rely a class-specific template to initialize our kinematic skeleton. Instead, we extract an approximate initial skeleton tree by Medial Axis Transform (MAT) and refine it during training.
Let M = [pm. |m = 1...., }} We choose the root of our kinematic model Proot as the medial axis point that is closest to all other medial-axis points, i.e., Proot == arg min E, lp -Pyl2, Vp, E M, i j.
} be the set of medial axis points extracted by applying a MAT on P

---

Next, we leverage dense sampling-gnn grid neighborhoods and define graph GM connecting neighboring points M. Points disconnected from Proot are removed. We then use a heuristic to select sparse joints (nodes) subset of Gar and define the bones (edges) based on their connectivity. To this extent, we traverse G starting from Proot breadth- first manner and mark Pm as a joint j. its geodesic distance from the preceding joint exceeds a threshold Blength 10. As result, we obtain a set lime-invariant canonical joint positions = [ which we further optimize during training.
While this skeleton is usually over-segmented, we show in our experiments that this does not hamper the training. We propose a pruning strategy to enable easier pose manipulation afterwards (see Fig.
Blend Skinning Weights For each point P. we initialize its raw skinning weight vector w, by an exponential decay function of the distance dist to each bone line by such that W = ( Before skinning, we obtain the final blend skinning weight vector w; through scaling by global learnable parameter and applying a softmax: Wi.j = =<"6s///S, /E ei,,// During the training, we optimize the initial W, as well as a. Accounting for the per-point weights, we define our full canonical feature point cloud as P {Pff. f,, wr)li N..
We forward- warp the canonical point cloud Pa an observation space of timestamp t via LBS [2]. The local transformation matrix T of each bone b is defined by a rotation RG around its parent joint ja- Consequently, each point p is transformed by linear combination of bone transformations as:
Point Warping
jo + RNJo"
P = TTPp
Σ Wi. wi TA with T = -T..T
Th. and To
(4)
1
where Tg is defined recursively by its parent bone Pa. Thp of the skeleton root is identity.
We express rotations Rg E SO(3) using the Rodrigues' formula, where f f= r/|[x|| the axis of rotation. However, learn the rotation angle directly find it leads to better pose regression than using 0 = ||r|| The time-dependent rotations rs, Og for each bone b are regressed by an MLP: (+)()) ri. 0f, ..., r 0f 6.... r'p, OB, , r Or,, tt, where r,. of and t, are the time-dependent root rotation and translation.
additional parameters because
Skeleton simplification The initial skeleton's segmentaiinnnn does not hamper pose recon- struction during training, but we optionally simplify the skeleton after training to ease pose editing (see Fig. Z). We prune merge bones based the size of rotation angles 0% produced by the transformation regressor Φ, Joints, which do not exhibit a rotational change from the rest pose above the threshold of deg in more than 5% of the observed timestamps,
marked static. We then merge the bones of such joints and their corresponding weights w. See the supplement for details.
4.3 Dynamic Point Rendering
We adopt the point cloud rendering from [21 but extend it to explicitly model rotational invariance of the radiance field. For each sampling point x, we find up to N 8 nearest observation feature points pf within radius of 0.01 and roto- aitassransl them into their canonical frames. This enables the feature embedding MLP P learn spatial relations in a pose-invariant invariant coordinate frame as:
f (f.. =f.),, with s/p (R) (x -p))).
(5)
where, RC is the rotation component T (Eq. 4 for point p and 7(-) is the positional encoding [
The neighboring embeddings f.., aggregated by inverse distance weighting, which produces the final feature input for Da and &. (see Eq, 2 and the consequent volume rendering (Eq. 3
N
d2
Ji with d; = =1pp - xl| : +a(()) c c (ef d).
(6)
dj
4.4 Losses
Next to the photometric loss Lphoto (Sec. B we utilize 2D chamfer loss to penalize the difference between the point cloud projected into a traiiin view [(P*) and the corresponding 2D ground truth
5

---

2 training
training
Watch-tt-Move
Ours
Watch-it-Move
Ours
Ground Truth
23.3 dB
29.4 dB
26.1
32.4 dB
21.8
26.8 dB
24.0 dB
28.7 dB
21.2 dB
27.5 dB
22.5 dB
28.6 de
Qualitative comparison displaying two held-out fiews-framess of scenes from the Robots rendered by WIM (20) and our method after 2 and 10 hours of training, and the PSNR scores.
Figure 3:
object mask M
Lmask Lohams f (I(P*), M The chamfer loss defined
Loham (Pt,p2)
Σ
min pp -P3 + P2 Σ
min [pp - -2313
(7)
PI
To prevent the skeleton from diverging too much from the medial axis M, we further minimize the chamfer loss between .M and the joints J (see Sec. 4.2). Lakee := Lcham y (M, JJ. In addition,
we minimize the transformation angles and the root translation: Ltrans = C2 1091) + |t Local
rigidity is further enforced upon points after deformation via
regularization:
N(d)
Larap ΣΣ
Σ
1p p31l2 1lpt 1/131-
Lastly, we apply two regularizers the blend skimning weights. To encourage smoothness, we penalize divergence of skinning weights in the rendering neighborhood N: Lsmooth
Σ. P EjeN(e) lw; Wyl. and, to encourage sparsity, we apply:
|Pd
Sippars ΣΣ way log(w),) + (1 wi log(1 wis)-
In total, our training loss is Wy Wn + W1 Lmas + WyCsket + Wy + Wy WrLemooth wi Esparse + ARAA where w (200, 0.02, 0.1, 10, 0.2, 0.005) in experiments.
5 Experiments
Here, we evaluate our work, which obtains state-of-the-art view-synthesis quality with lower training cost than other articulated methods. We also demonstrate class-agnostic reposing capability and we evaluate contribution of method components in ablation study. Video examples can be seen on the project website.
Datasets We chose three multi-vies video datasets that commonly used for the evaluation of dynamic multi-view synthesis methods and pose articulation. First, the Robots dataset [20] features multi-view synthetic videos of 8 topologically varied robots, making it well suited for testing pose articulation performance (see Fig. 44. We use 18 views for training and for evaluation. Second, the Blender dataset (3) is sparse multi-view synthetic dataset with 5 humanoid and 2 other articulated objects[] Its visual quality benchmarks are used to test the fidelity of image detail reconstruction (see
We do the multi-component Bouncing balls scene.

---

30
0.10
WIM Ours
Time Method
PSNR
SSIM1
LPIPS!
PSNR [dB]
0.08
WIM 20
25.05
0,952
0.059
25
2h
Ours
28.81
0.968
0.047
0.06
20
10h
WIM
27.61
0,964
0.046
Ours
30.19
0.973
0.041
10
10
Training time [hours]
Training time [hours]
Figure 4: Quality of unseen view synthesis during training with 95% confidence intervals in
Robots dataset [20). The initial plateau of WIM (20) matches the 10k warm-up steps used by the authors before training with all data. Our onset time corresponds to the 70 minutes required for pre-training of the backbone. Training of our method was terminated after 2.5 hours.
Method
Reposable
PSNR
SSIM
LPIPS
Training Time
D-NeRF (3
No
30.50
0.95
0.07
20 hours
Vox-B
No
32.67
0.97
0.04
28 mins
Hexplane
No
31.04
0.97
0.04
11.5 mins
K Plane hybrid
No
31.61
0.97
52min
WIM
Yes
23.81
0.91
0.10
hours
Ours
Yes
29.10
0.94
0.06
2.5 hours
Table 2: Quality of unseen view synthesis for the Blender dataset [3. Results of D-NeRF and
TiXeu reprinted from Fang et al.
"Without the "Bouncing balls" scene.
PSNR [del
34
0,980-
0,05
LPIPS
32-
0.04
0.970
0.03
30
10000
20000
10000 20000
10000
20000
Time Backbone [Iterations]
Time Backbone [Iterations]
Time Backbone [Iterations]
Figure 5: Effect of the backbone initialization pre mre-training steps the final result of our method when trained on the Jumping Jacks scene from the Blender dataset. Our final choice of 20k iterations corresponds to approximately 70 minutes of real time.
Fig. rright) We the original training-test split. Third, ZJU-MoCap dataset 82 is common test suite for dynamic human reconstruction and we evaluate 5 of its multi-view sequences with the same 6 training views used in Watch-Ii-Move (20). Finally, evaluate image quality using peak signal-to-noise ratio (PSNR), structural similarity (SSIM) 114, and learned perceptual image patch image metrics.
similarity (LPIPS) [155
Implementation details We pre-train the TiNeu Vox [I]] backbone using the authors' implementa- tion and an additional distortion loss [116). implemented in [117). Our method is implemented in Pytorch and we train each scene using the Adam optimizer for 160k (Blender and Robots) or 320k (ZJU-MoCap) iterations with batch size of 8192 rays, sampled randomly from multiple views and single timestamp. We choose the canonical timestamp by visual inspection, and gradually increase the number of observed timestamps during training. We adjust ray sampling and scheduling for the ZJU-MoCap dataset (see the Supplement). All experiments were done on a single Nvidia GPU RTX 3090Ti. See the supplementary materials for details. For more details, the Appendix.
Baselines We compare our method state-of-the of-the-artttttt non-articulated and articulated methods for high-fidelity multi-view video synthesis (see Table for an overview). D-NeRF [3] extends NeRF to the temporal domain by backward warping static canonical NeRF. TiNeuVox
[T]] improves
performance of D-NeRF using voxel grids. Hexplane (72) and K-Planes [11 decompose to the space-time volume to several hyper-planes. Finally, Watch-It-Move surface representation and LBS model for articulation. Note, that because aim at time-efficient
20 (WIM) jointly learns
7

---

WIM
Ours
Ground Truth
WIM
Ours
Ground Truth
Figure 6: While our model well reproduces details and non-rigidity of the Dinosaur (right), it can fail for complex motions combining long chains of rotations with texture cue ambiguity (left).
a) Learnd kinematic Model
bi Simpliifeed Kinernatic Model
Figure 7: Learned LBS weights and skeleton: a) After training. b) After additional post post-processing (weight merging and skeleton pruning, see Sec. 4.2)
learning, training of WIM was stopped after 11 hours (i.e., 80k of the original 400k optimization steps).
Novel view synthesis We provide results for the Robots view synthesis without the skeleton simplification; quantitatively in Fig. D and qualitatively in Fig. 3 More visual examples are available in the supplement. After the initial pre-training. our method quickly surpasses the image quality of WIM [20]. The mean image scores obtained for our method after 2 hours of training (inel. pre- training) are higher than those that WIM achieves after 10 hours. We attribute this to NeRF's capability to approximate even complex shapes, which are difficult to fit using the signed distance function utilized by WIM. Nevertheless, for visually simple objects with long nested pose transformation chains, this capacity might encourage false explanations of the articulation and cause artifacts, as visible in Fig. 6 left. However, this is not a common issue. We illustrate it in the Blender, where WIM struggles to represent fine details (see Fig. right), while our method achieves image scores close to those of non-reposable baselines (see Table 2 and the supplement).
Furthermore, Fig. shows that our image quality is not highly dependent on the pre-training phase. We observe that mere 100 iterations of TiNeu Vox pre-training provide an initialization sufficient for achieving PSNR scores over 30 dB by our method. However, while fine geometric details are still recovered, the coarse initial shape limits the skeleton complexity and, therefore, we opt for 20k pre-training steps in all other experiments.
ZJU-MoCap In Fig. 10 we compare our method to WIM in the ZJU-MoCap dataset. We observe that both methods are able to recover the 3D shape and the skeletal motion despite the difficulty of an accurate fine texture detail reconstruction. This can partly be attributed to imperfections in camera calibrations (see Supplement F of (96)) and partly by soft deformations of clothes which are modeled neither method. Notably, with a small modification our method learns to partially compensate for this. In Pose, we condition the feature embedding network 4 by the skeletal poses jointly learned from scratch by our method. This improves the image quality by modeling the residual deformations. See the Supplement for details. Finally, SM shows that replacing our skeleton in OursPose with a ground truth from an annotated SMPI. template (I5) does not affect the performance. This validates our skeleton initialization based on Medial Axis Transform.

---

Figure 8: Reposing using the simplified skeleton. Interpolation from canonical to novel pose.
a)
b)
Full Model
No Lasar
Full Model
Full Model
No Lyss
Ablation examples. a) CARAP enforces rigidity after pruning (see upper pipes), b) Lamooth results in better part-segmentation, c) Laket enforces the joints to stay inside the shape.
Figure 9:
Method
PSNR
SSIM
LPIPS
WIM 20
31.08
0.963
0.053
Ours
29.60
0.958
0,063
Ours
32.05
0.967
0.056
Ours SMPL
32.01
0.967
0.056
GT
WIM
Ours
GT
WIM
Ours
Figure 10: Comparison the /JU-MoCap dataset. Ours: Our full method. OursPose; Ours with an
additional pose-conditioned feature embedding network Φ, OursSMPL, OurgPose with ground truth SMPL 15 skeleton.
Reposing In Fig. 7 we visualize the learned blend- alend-skinning weights w, and skeletons with and without the skeleton simplification. The algorithm is able to substantially reduce the skeleton complexity, while largely preserving semantic articulations observed it is not able to remove all unnecessary skeletal branches when complex geometry is present (see Fig. right). In Fig- g we show that such post-processed skeletons allow for animating of novel poses by smoothly interpolating between user-defined poses. More animations can be found in the Supplement.
training data. However,
Loss ablation Our experiments suggest the regularization does improve the image quality, but it improves the quality of the kinematic model, which is important for our main goal of reposing (see Fig. 99. Specifically, CARAP avoids non-rigid distortions (a), Lsmooth leads more semantically meaningful part segmentation (b), Laker encourages functional placement of the skeleton joints inside object volume even for thin parts (c), and Liranf leads to reduction of necessary object parts after simplification (62 components are removed instead of 49 for Atlas). More details are provided the Supplement.
Backbone The TiNeuVox backbone allows us transfer parameters learned during the pre-training and initialize the features f; and the regressors & and Φ Interestingly, our experiment shows that such off-the- the-f features often do need any further fine-tuning (see Table Despite this, our method is agnostic to the backbone choice by design and the end-to-end training procedure of the entire model is essential for this. We demonstrate by training with random initialization. This way, only the point positions pf obtainable by any 3D reconstruction method are needed. Table 3 shows that this leads to only negligible drop in reconstruction quality in the Robots dataset. Here, the weight of Lyket (w2 was adjusted to prevent drift of the skeleton.

---

PSNR
SSIMI
LPIPS
PSNR
SSIM
LPIPS
Full Method 30.19
0.973
0.041
Full Method
29.10
0.94
0.06
Random Init. 29.97
0.971
0.045
No Fine-tuning 29.16
0.94
0.05
Table 3: Results with a TiNeu Vex initialization and with random initialization of the features f, and the regressors && and & in the Robots dataset.
Table 4: Results with and without fine-tuning of
the feature points f; in the Blender dataset.
Discussion
Limitations and Future Work We demonstrate fast learning of articulated NeRFs for state-of- the-art view synthesis and straightforward skeletal reposing for objects with piece-wise rigid pose transformations. While the LBS allows for fitting partially non-rigid deformations (see Fig. right), representing fully non-rigid, topologically varying, and multi-component objects would benefit from higher-dimennsional parameterization. Chen et al. [02 aim in that direction, but an intuitive and cost-effective reposing still favors skeleton-based techniques such as ours.
The performance of our method depends on the quality of the backbone reconstruction. While TiNeuVox [TT) supports reconstruction from even sparse data, a different backbone could offer more robust starting geometry and improve the skeleton initialization. Although our approach works well even for highly structured shapes and thin structures (see Fig. 6 right), we observe incorrect joint placement for objects with long chains of highly nonlinear geometrical transformations (see Fig. 6 left). Moreover, our approach successfully reduces the number of extraneous bones but is not able to completely eliminate all superfluous skeletal elements (see Fig. 7
Our method is restricted the kinematic motion space exhibited in the training sequences. While extrapolation is possible to some extent, the proposed method is not able to generalize to arbitrary unseen poses. Finally, we focus on image synthesis for user-defined poses rather than a direct fitting of unseen skeletal poses from images or motion capture. An inverse fitting of skeletal poses to novel 2D observations or transfer of poses from one object to another remain opportunities for future research.
Conclusion We presented a method for fast learning of articulated models for high- novel view synthesis of captured 3D objects. Our aeward-warped neural point clouds avoid the pitfalls of backward warping and elegantly integrate skeleton-based LBS. As result, our method merges straightforward reposing even of strongly animated inputs, present in the Robots dataset, with high visual fidelity of NeRF rendering. Our work is significant step towards low-cost acquisition of animatable 3D objects for games, movies, and education.
Ethical Considerations Our method renders novel views and poses of 3D objects including human bodies. However, we do not focus on this class and we note that many human-specific methods exist (see Sec. Nevertheless, we acknowledge that our method could potentially be used to produce fake images of people and that research is needed to understand the risks and their mitigation.
10

---

References
[1] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
[2] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. Advances in neural rendering. In Computer Graphics Forum, volume 41, pages 703-735. Wiley Online Library, 2022.
[3] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. arXiv preprint (rXiv22011.1396/ 2020.
[4] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.
[5] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Thcobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis dynamic scene from monocular video. In ICCV, 2021.
[6] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan Goldman, Steven M. Seitz, and Ricardo Martin-Brualla, Nerfies: Deformable neural radiance fields. ICCV, 2021.
[7] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In CVPR, pages 9421-9431, 2021.
[8] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In CVPR, 2021.
[9] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin- Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph. (SIGGRAPH Asia), 40(6), 2021.
[10] Tianye Li, Mira Slaveheva, Michael Zollhöfer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, and Zhaoyang Lv. Neural 3d video syathesis from multi-view video. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 5521-5531, June 2022.
[11] Jiemin Fang, Taoran Yi, Xinggang Wang. Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
[12] John Lewis, Matt Cordner, and Nickson Fong. Pose space deformation: a unified appreach to shape interpolation and graphics and interactive techniques, pages 165-172, 2000.
deformation. In Proceedings of the 27th annual conference on Computer
[13] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In ACM SIGGRAPHI page 187-194, 1999.
[14] Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. I.carning a model of facial shape and expression from 4D scans. ACM Trans. Graph. (SIGGRAPH Asia), 16:6))19: 2017.
[15] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (SIGGRAPH Asia), 3466)224 2015.
[16] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black, Expressive body capture: 3D hands, face, and body from a single image. In CVPR, pages 10975-10985, 2019.
[17] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T Freeman, and Ce Liu. Lasr: Learning articulated shape reconstruction from a monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15980-15989, 2021.
[18] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu, and Deva Ramanan. Viser: Video-specific surface embeddings for articulated 3d shape reconstruction. Advances in Neural Information Processing Systems, 34:19326 11926-1933888. 2021.

---

[19] Chun-Han Yao. Wei-Chih Hung. Yuanzhen Li, Michael Rubinstein. Ming-Hsuan Yang. and Varun Jampani. Lassie: Learning articulated shapes from sparse image ensemble via 3d part discovery. arXiv preprint ar0iv 004344, 2022.
[20] Atsuhiro Noguchi, Umar lqbal, Jonathan Tremblay. Tatsuya Harada, and Orazio Gallo. Watch it move: Unsupervised discovery 3d joints for re-posing of articulated objects. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3677-3687, 2022.
[21] Qiangeng Xu, Zexiang Xu, Julien Philip. Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point- bastd neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5438 5448, 2022.
[22] Gengshan Yang, Minh Vo, Neverova Natalia, Deva Ramanan, Vedaldi Andrea, and Joo Hanbyul. Banmo: Building animatable 3d neural models from many casual videos. In CVPR, 2022.
[23] Yuefan Wu", Zeyuan Chen Shaowei Liu, Zhongzheng Ren, and Shenlong Wang. CASA: Category- agnostic skeletal animal reconstruction. Neural Information Processing Systems (NeurIPS), 2022.
[24] SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari Morcos, Marta Gamelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et Neural scene representation and rendering. Science, 2018.
[25] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[26] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occu- pancy networks: Learning 3D reconstruction in function space. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[27] Mateusz Michalkiewicz, Jhony K Pontes, Dominie Jack, Mahsa Baktashmotlagh, and Anders Friksson. Implicit surface representations as layers neural networks. In IEFE International Conference on Computer Vision (ICCV), 2019.
[28] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[29] Matan Atzmon and Yaron Lipman. SAL:: Sign agnostic learning of shapes from raw data. In IEEE Conference on Computer Vision and Puttern Recugnition (CVPR), 2020.
[30] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In International Conference on Machine Learning (ICML), 2020.
[31] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nicßner, and Thomas Funkhouser. Local implicit grid representations for 3D scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[32] Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. Overfit neural networks as a compact shape representation. arXiv preprins arXiv:2009.09800 2020.
[33] Rohan Chabra, Jan Eric Lenssen, Eddy 1lg. Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In European Conference Computer Vision (ECCV), 2020.
[34] Songyou Peng, Michael Niemeyer, Lars Mcscheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference Computer Vision (ECCV), 2020.
[35] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahraa,, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[36] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Advances Neural Information Processing Systems (NeurIPS), 2020.
[37] Julien N.P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. ACORN: Adaptive coordinate networks for neural representation. ACM Transactions on Graphics (SIGGRAPH), 2021.
12

---

[38] Petr Kellnhofer, Lars Jebe. Andrew Jones, Ryan Spicer. Kari Pulli, and Gordon Wetzstein. Neural lumigraph rendering. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[39] Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. SDFDiff: Differentiable rendering of signed distance fields for 3D shape optimization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[40] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[41] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous DD-structure-aver aware neural scene representations. In Advances in Neural Information Processing Systems (NeurlPS), 2019.
[42] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Weizstein, and Michael Zollhöfer. Voxels: Learning persistent 3D feature embeddings. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019. 2019.
[43] Shunsuke Saito, Zong Huang, Ryota Natsume, Shigco Morishima, Angjoo Kanazawa, and Hao Li. PfFu: Pixel-aligned implicit function for righuucsotution clothed human digitization. In /EEE International Conference on Computer Vision (ICCV), 2019.
[44] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3D supervision. arXiv preprint arXiv:191 100e:/911.0000000 1,00667, 2019.
[45] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Transactions Graphics (SIGGRAPH). 2019.
[46] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[47] Shaohui Liu, Yinda Zhang, Songyou Peng. Boxin Shi, Mare Pollefeys, and Zhaopeng Cui. DIST: Rendering deep implicit signed distance function with differentiahle sphere tracing. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[48] Ricardo Martin Bruuala, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[49] Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. NeRV: Neural reflectance and visibility fields for relighting and view synthesis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[50] Kai Zhang. Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
[51] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H. Mueller, Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, and Markus Steinberger. DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks. Computer Graphics Forum, 40(4), 2021.
[52] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[53] David B Lindell, Julien NP Martel, and Gordon Wetzstein. AutoInt: Automatic integration for fast neural volume rendering. JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[54] Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind. Unconstrained scene generation with locally conditioned radiance fields. arXiv preprint ar 2021.
[55] Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: Unifying neural implicit surfaces and radiance fields for vie reconstruction. In IEEE International Conference on Computer Vision (ICCV), 2021.
13

---

[56] Stephan J Garbin, Marek Kowalski, Malthew Johnson, Jamie Shotton, and Julien Valentin. FastNeRF: High-fidelity neural rendering at 200fps. arXiv preprini arXiv:2103. 2021.
[57] Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, et al. State of the art on neural rendering. Farographics Association, 2020.
[58] Christian Reiser, Songyou Peng. Yiyi Liao, and Andreas Geiger, KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs. In IEEE International Conference Computer Vision (ICCV), 2021.
[59] Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. In Proceedings of the IEEE/CVF Conference Compuder Vision and Pattern Recognition, pages 14153-14161, 2021.
[60] Matthew Tancik, Vincent Casser Xinehen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-NeRF: Scalable large seene neural view synthesis. arXiv, 2022.
[61] Peter Hedman. Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Debevec. Baking neural radiance fields for real-tine saal-tmme view synthesis. In IEEE International Conference on Computer Vision (ICCV), 2021.
[62] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In IEEE International Conference on Computer Vision (ICCV), 2021.
[63] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. arXiv: 055999 January 2022.
[64] Cheng Sun, Min Sun, and Hwann-Tzoon Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459-5469, 2022.
[65] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theohalt. Neural sparse voxel fields. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[66] Julian Ost, Issam Laradji, Alejandro Newell, Yuval Bahat, and Felix Heide. Neural point light fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18419- 18429, 2022.
[67] Jad Abou Abou-Chakra, Feras Dayoub, and Niko Sünderhauf, Particlenerf: Particle based encoding for online neural radiance fields in dynamic scenes. arXiv preprint arXiv:2211.04044 2022.
[68] Ruofan Liang, Jiahao Zhang, Haoda Li, Chen Yang, and Nandita Vijaykumar, Spidr: Sdf-based neural point fields for illumination and deformation. arXiv preprint arXiv:2210.08390 2022.
[69] Yuteng Zheng, Wang Yifan, Gordon Wetzstein, Michael Black, and Otmar Hilliges. Pointavatar: Deformable paset-based head avatars from videos. arXiv preprint arXiv:2212.0837 2022.
[70] Shengze Wang, Alexey Supikov, Joshua Rateliff, Henry Fuchs, and Ronald Azuma. Inv: Towards streaming incremental neural videos. arXiv preprint arXiv:2302.01532 2023.
[71] Sara -ridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12488, 2023.
[72] Ang Cao and Justin Johnson. Hexplane: fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 130-141, 2023.
[73] Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Trans. Graph. (SIGGRAPH Asia), 36(6), November 2017.
[74] Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black. 3d menagerie: Modeling the 3d shape and pose of animals. In CVPR, pages 6365-6373, 2017.
[75] Guy Gafni, Justus Thics, Michael Zollhöfer, and Matthias Nicßner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In CVPR, 2021.
[76] Ziyan Wang. Timur Bagautdinov, Stephen Lombardi, Tomas Simon, Jason Saragih, Jessica Hodgins, and Michael Zollhofer. Learning compositional radiance fields of dynamic human heads. In CVPR, pages 5704-574 5713, 2021.
14

---

[77] Yudong Guo, Keyu Chen, Sen Liang, Yongjin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In ICCV, 2021.
[78] Yuteng Zheng, Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, and Otmar Hilliges. M Avatar: Implicit morphable head avatars from videos. Computer Vision and Pattern Recognition (CVPR), 2022.
[79] Kacper Kania, Kwang Moo Yi, Marek Kowalski, Tomasz Trzcińskii, and Andrea Tagliasacchi. CoNeRE: Controllable Neural Radiance Fields. In CVPR, 2022.
[80] Yiyu Zhuang, Hao Zhu, Xusen Sun, and Xun Cao. Mofanerf: Morphable facial neural radiance field. In Computer Vision-EEC ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part III, pages 268-285. Springer, 2022.
[81] ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu. Rignerf: Fully controllable neural 3d portraits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 20364-20373, 2022.
[82] Sida Peng. Yuanqing Zhang, Yinghao Xu. Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021.
[83] Sida Peng, Junting Dong, Qianqian Wang. Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In ICCV, 2021.
[84] Shih- Yang Su, Frank Yu, Michael Zollhöfer. and Helge Rhodin. A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. In NeurlPS, 2021.
[85] Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. In NeurIPS, 2021.
[86] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. In ICCV, 2021.
[87] Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural human performer: Learning generalizable radiance fields for human performance rendering. In NeurlPS, 2021.
[88] Haotong Lin, Sida Peng, Zhen Xu, Hujum Bao, and Xiaowei Zhou. Efficient neural radiance fields with learned depth-guided sampling. In arXiv, 2021.
[89] Tao Hu, Tao Yu, Zerong Zheng, He Zhang. Yebin Liu, and Matthias Zwicker. Hvtr: Hybrid volumetric- textural rendering for human avatars. In arXiv, 2021.
[90] Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Thcobalt. Neural actor: Neural free-view synthesis of human actors with pose control. ACM Trans. Craph.(ACM SIGGRAPH Asia), 2021.
[91] Tianhan Xu, Yasuhiro Fujita, and Ficchi Matsumoto. Surface-aligned neural radiance fields for controllable 3d human synthesis. In CVPR, 2022.
[92] Fugiang Zhao, Wei Yang. Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, and Lan Xu. Human- nerf: Efficiently generated human radiance field from sparse imputs In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7743-7777777 14437773, 2022.
[93] Chung-Yi Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, and Ira Kemelmacher- Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In CVPR, 2022.
[94] Boyi Jiang, Yang Hong, Hujun Bao, and Juyong Zhang. Selfrecon: Self reconstruction your digital avatar from monocular video. In CVPR, 2022.
[95] Xu Chen, Yufeng Zheng, Michael Black, Otmar Hilliges, and Andreas Geiger. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11594-11604, 2021.
[96] Ruilong Li. Julian Tanke, Minh Vo, Michael Zollhöfer. Jürgen Gall, Angioo Kanazawa, and Christoph Lassner. Tava: Template-fier freee animatable volumetric actors. In European Conference on Computer Vision, pages 419 436. Springer, 2022.
15

---

[97] Yihao Zhi, Shenhan Qian, Xinhao Yan, and Shenghua Gao. Dual-space nerf: Learning animatable avatars and scene lighting in separate spaces. In 2022 International Conference 3D Vision (3DV). pages 1-10. IEEE, 2022.
[98] Yang Hong, Bo Peng, Haiyao Xiao. Ligang Liu, and Juyong Zhang. Headnerf: A real-time berf-based parametric head model. In CVPR, 2022.
[99] Alexander W Bergman, P Kellnhofer, Wang Yifan, Eric R Chan, David B Lindell, and Gordon Wetzstein. Generative neural articulated radiance fields. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022): NeurlPS Proceedings. 2022.
[100] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Unsupervised learning of efficient geometry-awar neural articulated representations. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XVII. pages 614. Springer, 2022.
[101] Suyi Jiang, Haoran Jiang, Ziyu Wang, Haimin Luo, Wenzheng Chen, and Lan Xu. Humangen: Generating human radiance fields with explicit priors. arXiv preprint arX1v:2212..05321, 2022.
[102] Hsiao-yu Chen, Edith Tretschk, Tuur Stuyck, Petr Kadlecek, Ladislav Kavan, Etienne Vouga, and Christoph Lassner. Virtual elastic objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15827-15833 2022.
[103] Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Magicpony: Learning articulated 3d animals in the wild. arXiv preprint urXiv:2211.22445 124997, 2022.
[104] Angjoo Kanazawa, Michael Black. David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7122-7131, 2018. 2018.
[105] Nikos Kolotouros, Georgios Pavlakos, Michael Black, and Kostas Danillidis. Learning reconstruct 3d human pose and shape via model-fitting in the loop. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 2252 22212, 2019.
[106] Soubhik Sanyal, Timo Bolkart, Haiwen Feng. and Michael J Black. Learning regress 3d face shape and expression from an image without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7772-777,, 2019.
[107] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned implicit function for high-rosolution 3d human digitization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 84 93, 2020.
[108] Muhammed Kocabas, Nikos Athanasiou, and Michael Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9233-5263, 2020.
[109] Chun-Han Yao, Wei-Chih Hung. Yuanzhen Li, Michael Rubinstein. Ming-Hsuan Yang. and Varun Jampani. Hi-lassie: High-fidelity articulated shape and skeleton discovery from sparse image ensemble. arXiv preprint arXiv:2212. 1042, 2022.
[110] Tianhan Xu and Tatsuya Harada. Deforming radiance fields with cages. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXIII, pages 159-175. Springer, 2022.
[111] Kaizhi Yang, Xiaoshuai Zhang, Zhiao Huang, Xuejin Chen, Zexiang Xu, and Hao Su. Movingparts: Motion-based 3d part discovery in dynamic radiance field. arviv preprint 23X::2303300703 2023.
[112] James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics, 18(3):165- 174, 1984.
[113] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In Computer Vision-ECC ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Par XXXII, pages 333-350. Springer, 2022.
[114] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 1 6104)2600-12,, 2004.
[115] Richard Zhang. Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features a perceptual metric. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 386-595, 2018.
16

---

[116] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[117] Cheng Sun, Min Sun, and Hwann- Hwann-Tzong Tzong Chen. Improved direct voxel grid optimization for radiance fields reconstruction. arXiv preprint arXiv:2206.05085 2022.
17


Reviews:
[{"review_id": "BTmW7WuwjL", "reviewer": "Reviewer_KBgL", "summary": "This paper tackles the task of dynamic novel view synthesis from multiview videos and aims for the ability of reposing. It tackles the problem with a point-based rendering approach. More importantly, it does not need any pre-defined or class-specific template/skeletons and learns a per-video data-driven skeleton. The effectiveness is verified on two datasets, i.e., Robots and Blender.", "questions": "See above.", "limitations": "The authors provide a discussion on limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n\n2. The proposed skeleton learning is quite interesting: using a data-driven point cloud from TiNeuVox; then approximating an initial skeleton via Medial Axis Transform (MAT); finally using another data-driven component to refine the skeleton from MAT.", "weaknesses": "### 1. Robustness to Initialization\n\na. The proposed approach heavily relies on the NeRF pretraining: 1) geometry-wise, the initial skeleton needs the pre-trained NeRF's density function to sample points, which will be later used to run MAT; 2) appearance-wise, the points' features come from the pre-trained NeRF.\n\nb. A natural question is how robust the proposed approach is to the initialization. One way to demonstrate this could be: 1) train TiNeuVox until convergence; 2) initialize the proposed approach from different checkpoints of TiNeuVox's training; 3) draw a plot of the final performance wrt different initializations.\n\nc. What is related is: can authors clarify what the statement \"we uniformly sample the density $\\sigma$\" (L124) means? Does it mean that you will first compute densities on plenty of points to get a sense of min and max densities? And how will you sample the point cloud with such min/max values?\n\n### 2. Evaluation of Human Dataset\n\nArguably, one of the most important articulated view synthesis scenarios is human-related rendering. I am wondering whether authors can provide quantitative and qualitative results for the proposed approach on some human-related datasets, e.g., ZJU-MoCap used in [20]. Such results could provide a more complete assessment of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback! We answer your specific questions in this document. Please refer to the shared response for the experiment results and answers to common questions.\n\n**Robustness to Initialization**\n\nWe conducted the proposed experiment, and we report the results in the shared response. They show that our method is robust to the quality of the initial geometry as long as functional parts are recovered. \n\n**Clarification Density Sampling**\n\nWe will clarify that we sample the canonical density function $\\Phi_d$ of TiNeuVox on a uniform coordinate grid and discard empty samples through thresholding. The grid resolution is adaptively chosen to retain approximately 10k points.\n\n**Human-Related Rendering**\n\nWe tested our method with the ZJU-Mocap human capture dataset. Please refer to the shared section and the attached PDF for details.\n\n**Reliance on Pre-training**\n\nWe included an additional experiment which shows that our method is robust to the choice of the backbone-specific initialization of the features and decoders. Please refer to the shared response \"Impact of the Backbone Choice\" for details.\n"}, {"review_id": "m7fTjH3Y7N", "reviewer": "Reviewer_ZYjY", "summary": "The paper presents an approach to articulated view synthesis, introducing the concept of Template-free Articulated Neural Point Clouds. The authors utilize a structure-free point-based NeRF representation which supports forward-warping of canonical objects to any poses through Linear Blend Skinning (LBS). Such a representation allows for the joint optimization of LBS pose parameters as well as dynamic NeRFs in a short training time. The method is evaluated on two datasets and compared with existing methods, such as D-NeRF, TiNeuVox, and WIM, showing superior or comparable performance on dynamic novel view synthesis and novel pose synthesis.", "questions": "* Please refer to the weaknesses section.\n\n* How to optimize blend skinning weight vector $w_i$? And what is the motivation for defining another $\\alpha$ for scaling the weights?\n\n* $\\Phi_d$ should be $\\Phi_b$ in line 95 to represent the backward-warping MLP.", "limitations": "* The authors could discuss how variations in image quality, such as resolution, lighting conditions, and the presence of noise, might impact the performance of their method.\n\n* How well does the method perform when applied to longer sequences or more complex scenes?", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(+) The paper provides thorough explanations of the model and experiments.\n\n(+) The paper proposed an effective point-based NeRF representation to support the forward-warping of the canonical space for template-free objects.\n\n(+) The method achieves better novel view synthesis and reposing results in a shorter training time on two datasets compared to existing approaches. The visualized learned LBS weights fields and poses are cleaner than WIM.", "weaknesses": "(-) Inconsistent Motivation: The paper's motivation appears to be inconsistent throughout the text. Initially, the primary motivations are identified as 1) **Reposability** and 2) **Efficiency**. However, in the model section, the method is built upon a pre-trained TiNeuVox as initialization, aiming to animate TiNeuVox to new poses. Consequently, the efficiency of the proposed method heavily depends on the backbone/representation used, and the main motivation here seems to be **Reposability**. In the experimental section, the method primarily compares with WIM to demonstrate its efficiency over WIM on novel view/pose synthesis. A more convincing demonstration of the **Reposability** performance of the proposed method would involve comparing it with WIM in __its original training iterations__. To prove the **Efficiency** of the method, it would be beneficial to investigate the influence of different backbones/representations on the proposed method.\n\n(-) Performance and Efficiency: The performance and efficiency of the proposed method do not seem promising compared to TiNeuVox. Given that the method builds upon a pre-trained TiNeuVox as initialization and aims to animate TiNeuVox to new poses, one would expect the method to perform at least similarly to TiNeuVox on the novel view synthesis task. However, as shown in Table 1 of the supplementary material, the method performs worse than TiNeuVox on 5 out of 7 scenes and requires significantly longer training time.\n\n(-) Dependence on Pre-trained Model: The proposed method relies on a pre-trained model as initialization, whereas WIM does not require such a pre-trained model. For a fair comparison, it would be advisable to also compare with a modified version of WIM that is initialized from a pre-trained model. This comparison would provide a more balanced view of the strengths and weaknesses of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback! We answer your specific questions in this document. Please refer to the shared response for the experiment results and for answers to common questions.\n\n**Inconsistent Motivation & Comparison to WIM with its Original Training Iterations**\n\nWe added additional experiments showing that our methods can work with different backbones. Please refer to the shared response.\nAs described in our paper, we modified the original scheduler of the WIM to access the full training data after the initialization phase. This was required because the original scheduler would access the full training sequence for the initial 80K iterations (around 11 hours with our GPU). This would invalidate the training curves in Figure 4. \n\nTo demonstrate that this does not degrade WIM’s performance, we completed the full original training schedule with 200K iterations (~28h) for the robot ‘Spot’ using our data split. While it did eventually achieve a higher PSNR of 27.9 dB it did not get anywhere close to our 32.4 dB. Furthermore, it did this with almost 3x training time compared to our own method. The same holds for the second “merge” phase of WIM (a total of 350K iterations, ~ 2 days) which results in PSNR of 28.2 dB. \n\nImportantly, the PSNR of 25.9 dB achieved by WIM with its original scheduler after 80K steps (~11h) is comparable to that of our modified scheduler (26.3 dB) after the same number of iterations. This demonstrates that our comparison setup is fair.\n\nPlease note that full training of WIM for multiple scenes is not possible within the rebuttal period which further highlights the difference in efficiency. Moreover, we were forced to reduce the batch size from 16 to 8 frames to fit into the 24 GiB VRAM of our RTX 3090 GPU.\n\n**Performance and Efficiency**\n\nUnlike TiNeuVox and other general dynamic NeRFs, our representation follows the Linear Blend Skinning which reduces the dimensionality of the pose space to enable reposing. This inevitably restricts the freedom for fitting the residual deformations that do not follow the low-dimensional model. We believe that explains the gap between the reposable and the non-reposable methods in Table 1 of the paper Supplement. \n\nFurthermore, the canonical volume in TiNeuVox is time-conditioned and, hence, it can lead to temporal changes not described by the deformation field alone [0]. This has been reported to further reduce the fitting error [1] but it goes against our idea of disentanglement between the canonical representation and reposing.\n\nWe will make this distinction clear and highlight that our method achieves consistently better performance than the other reposable baseline, WIM. We believe that future research based on our method will bridge the gap by explicitly modeling the additional scene changes as pose-conditioned residuals.\n\n**Dependence on Pre-trained Models**\n\nWhile we agree that it would be interesting to study a similar initialization for other methods including Watch-It-Move, we note that the design of such an experiment is not obvious. The authors do not propose any way to initialize the representation based on a known density distribution and a trivial modification is not possible because, unlike our method, the Watch-It-Move representation is implicit. Moreover, it features an inherent ambiguity between the per-part ellipsoids $SDF_i(.)$ and the residual SDF MLP $S_\\Theta(.)$. We argue that such a study is outside of the scope of the paper as we primarily focus on learning reposability.\nTo demonstrate that our method is not closely tied to TiNeuVox, we conducted an experiment where we only initialize positions of the feature points but leave all feature values and network parameters random like in [3]. This partially approximates the behavior of WIM which spends the first 10K iterations to recover a coarse static representation from the first few sequence frames. Please refer to the shared response and the attached PDF.\n\n**Blend Skinning Vector and Alpha Scaling**\n\nThe blend skinning vectors $w_i$ are optimized jointly end-to-end. The $\\alpha$ parameter maps linear point-to-bone distances to a non-linear space, similarly as in [4]. Because $\\alpha$ is global and shared by all blend skinning weights for all bones, it distributes gradient updates across the LBS model and reduces within-part variation of the effective skinning weights. \n\n**Impact of Dataset Image Quality, Length and Complexity**\n\nWe conducted an additional experiment with the ZJU-Mocap dataset [5]. This is a challenging dataset because it contains longer and more realistic motion sequences and also realistic captured appearance, noise and complex lighting. Please refer to the shared response for details and results.\n\n**Additional comments**\n\nWe will correct the typo in line 95. \n\n**References**\n\n[0]: Tretschk, Edith, et al. \"State of the Art in Dense Monocular Non‐Rigid 3D Reconstruction.\" Computer Graphics Forum. Vol. 42. No. 2. 2023.\n\n[1]: Liu, Yu-Lun, et al. \"Robust dynamic radiance fields.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[3]: Xu, Qiangeng, et al. \"Point-nerf: Point-based neural radiance fields.\" Proceedings of the IEEE/CVF CVPR 2022.\n\n[4]: Yang, Gengshan, et al. \"Viser: Video-specific surface embeddings for articulated 3d shape reconstruction.\" Advances in Neural Information Processing Systems 34 (2021): 19326-19338.\n\n[5] Peng, Sida, et al. \"Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.\" Proceedings of the IEEE/CVF CVPR 2021.\n\n[6] Li, Ruilong, et al. \"Tava: Template-free animatable volumetric actors.\" ECCV 2022.\n"}, {"review_id": "8gVGRpUhYN", "reviewer": "Reviewer_bAZS", "summary": "The method reconstructs a reposable Dynamic NeRF of an articulated object from multiview videos. This is achieved by using linear blend skinning (LBS) of an automatically extracted skeleton to represent the deformation from canonical to observation space. ", "questions": "Thank you for the supplemental video. I think the Ground Truth T-Rex needs to also be animated.\n\nLine 95 mentions Fi_d as a backward-warping MLP, did you mean Fi_b?\n\nLine 98 uses p' in an expression for v_m, was that supposed to be p^c?\n\nSpecific descriptions of various MLPs listed in line 101 would be helpful.\n\nIn line 103, did you mean to use delta_i instead of just delta?\n\nIs the N in equation 8 (L_arap) the same N as in L_smooth?\n\nHow come the simplified skeletons for the man and dinosaur are so sparse, compared to the two robots?", "limitations": "The authors address the main limitations of their approach, namely that LBS restricts the method to individual rigidly-linked bodies (as opposed to deformable objects or full scenes).", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Combining LBS kinematics with NeRF appearance model effectively produces compelling reposing results.\nYielding a posable skeleton enables numerous traditional animation workflows.", "weaknesses": "A quick literature search yielded https://arxiv.org/abs/2208.14851 which talks about reposing with a different way of avoiding inverse mapping issues. It may warrant a mention in related work. Could that approach be combined with this paper's automatic skeleton extraction (they use a known rigged template mesh) to yield similar results?\n \nGiven that the method yields a posable skeleton, the results could have been much more compelling with artist-made or even mocap-driven animations (rather than just blending between a couple of poses).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback! We answer your specific questions in this document. Please refer to the shared response for the experiment results and for answers to common questions.\n\n**Relation to Dual-Space NeRF**\n\nThank you for the suggestion, we will discuss Dual-Space NeRF [0] as an additional example of a template-based method for human bodies as it relies on the SMPL model for reposing. We note that our method differently targets general template-free reposing.\n\n**Reposable Skeleton Animation**\n\nWe agree that making our pose representation more artist-friendly for example by directly applying motion captured data is an interesting open problem. We believe that future work in this direction will benefit from our contribution and build additional interfaces on top of it to enable intuitive manipulation of general articulated objects. \n\n**Reposed T-Rex**\n\nWe have added an example showing a user-driven opening of the T-Rex’s mouth. Please refer to the PDF attachment Figure 2.\n\n**Specific descriptions of various MLPs**\n\nThank you for the suggestion, we will add a table specifying the different MLPs in the appendix. We will also make our code publicly available upon acceptance.\n\n**Sparsity of Skeletons**\n\nOur Post-processing step prunes skeleton bones that are not needed to represent motion observed in the input video sequence. Therefore, for sequences with less motion, the skeleton may be reduced to only a few bones. This is the case for the T-Rex scene, for example, which only tilts the full body upward and opens its mouth.  We note that this is not a limitation but an intentional design feature that reduces the number of degrees of freedom for a potential animator, and that this step can be omitted if desired.\n\n**Other comments**\n\nWe will correct the issues in lines 95, 98, and 103. The N in Eq. 8 and in $\\mathcal{L}_\\textrm{smooth}$ are indeed identical even though different neighborhood sizes could be considered if required.\n\n**References**\n\n[0] Zhi, Yihao, et al. \"Dual-space nerf: Learning animatable avatars and scene lighting in separate spaces.\" 2022 International Conference on 3D Vision (3DV). IEEE, 2022.\n"}, {"review_id": "uvnYDjBwl6", "reviewer": "Reviewer_RMHs", "summary": "Authors presents a method to learn articulated model from multi-view video. And demonstrate an ability for efficient learning skeleton pose along as view synthesis model for dynamical structures.  Moreover the suggested method drastically improve convergence compared to naive approaches. To extract the skeleton authors apply Medial Axis Transform for initalization and RBF weights for the joints.\n The results on the D-NeRF dataset are demonstrate the superiority of this model over predecessor.", "questions": "Why do you need  time-dependent rotations?\nDoes your training time include pre-training of the dynamic scene in Tab 2.\nDo you need ARAP and transformation loss, how do they influence on the training.\nHave you consider using something different to TiNeuVox method, there are plenty of much faster and accurate versions (e.g. HexPlane, Tensor4D)?", "limitations": "Highly relying on the retrained method\nSpeed and memory consumption for higher resolution \nSimplification and skeleton construction itself is based on heuristic and can be very poor on some scenes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "I appreciate the ideas in the method especially skeleton construction and mask loss.\n- The approach doesn’t take into account pre-defined joint structure\n- Convergence is quite fast\n- Can handle non-rigid motions (in theory) \n- This a first method learning articulation model purely from the input images\nThe description of the method is clear and solid.", "weaknesses": "The main idea is interesting, but the comparison is poor. \nThe idea of skelton initializing is mostly based on the previous works, and the it is not clear how much the final quality degrades, it is necessary to compare it with the ideal human skeleton, for example.\nTo apply this technique you first have to pre-train a dynamic version of NeRF, that complicates method compared to others. \nAll experiments are done for a single backbone and extremely limited variety of data\nAs you mentioned this method assume single object in the centre of the scene.\nSimplified model produces unrealistic results (figure 6) the skeleton even cannot cover actions from training.\nThere are some works (e.g. TAVA) that incorporate idea of non-template methods (fairly they utilize keypoints) that have to be mentioned\nHuman related baselines can be interesting here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback! We answer your specific questions in this document. Please refer to the shared response for the experiment results and for answers to common questions.\n\n**Pre-trained Dynamic NeRF and Single Backbone**\n\nWe have added additional results showing that our method can work with arbitrary backbones, if they yield a point cloud. Therefore, an initialization could also be achieved by a static pre-trained NeRFs or other means as in [0]. However, in our use-case we chose TiNeuVox because it can handle settings where a single view at a given time step is available which is the case for the Blender data set.\n\n**Comparison to TAVA**\n\nThank you for pointing out TAVA [1]. We will discuss it as an additional example of an articulated reconstruction method. While TAVA utilizes keypoints instead of a full template, unlike our method, it still needs a tracked skeleton as an additional input. In our experiments we focus on comparisons to Watch-It-Move as the closest baseline because, like us, it does not need any prior skeletal information.\n\n**Need for Time-Dependent Rotations**\n\nOur method applies an Analysis-by-Synthesis approach. Therefore, we must learn the time-dependent rotations to warp the canonical representation to a time-specific representation to reconstruct the training images. Unlike TAVA we do not have access to tracked skeletons.\n\n**Does your Training Time Include Pre-Training of the Dynamic Scene?**\n\nYes, all reported times do include the pre-training which is visualized by the horizontal offset of starting points for the blue “Ours” curves in Figure 4.\n\n**ARAP and Transformation Loss**\n\nARAP enforces per-part rigidity by avoiding excessive weight mixing (please refer to Figure 8a in the main paper). Transformation loss enforces sparsity of joint rotations which helps with pruning in the post-processing step (please refer to Section 5 in the supplements)\n\n**Alternative backbones**\n\nYes, please refer to the additional results where we show that random point initialization suffices for our method, therefore, Hexplane and Tensor4D could be utilized, too.\n\n**References**\n\n[0]: Xu, Qiangeng, et al. \"Point-nerf: Point-based neural radiance fields.\" Proceedings of the IEEE/CVF CVPR 2022.\n\n[1] Li, Ruilong, et al. \"Tava: Template-free animatable volumetric actors.\" ECCV 2022.\n"}, {"review_id": "RSQxw4KaqK", "reviewer": "Reviewer_u7Dh", "summary": "The authors present a method for dynamic radiance fields of subjects whose motion can be described by skeletal animation. The method automatically extracts a skeleton using medial axis transform. Further it obtains a object feature point cloud from a pre-trained NeRF and expresses their position as a function of the skeleton via linear blend skinning.\nThe animatable point cloud (including skinning weights and animation) is optimized using the PointNeRF renderer using losses for RGB, rigidity, animation smoothness and 2D CD against the mask. After optimization, it can be manually reposed and rendered from novel views.\n\nThe method is evaluated on the Robots and Blender datasets where it compares favorably against previous work.\n\n", "questions": "It does not become clear to me what exactly is done with the features $\\mathbf{f}_i$ coming from TiNeuVox. Are they just used as point feature initialization and then further optimized using PointNeRF or are they kept fixed? I can imagine that the PointNeRF renderer could adapt to these features without modifying them. This boils down to the question of which parameters of the model are optimized when, which seems to be not completely clear in the paper.\n", "limitations": "Limitations and potential negative societal impact are discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Using a point based representation to solve the given task is a good idea and makes sense, given that deformations might be very local and not restricted to a smaller set of rigid parts.\n- The ability for automatic part decomposition and reposing are interesting contributions that adds value with respect to other dynamic NeRFs that might produce better visual results.\n- The method compares favorably against the most relevant previous work WIM.\n- The extracted part decompositions and skinning weights look very good and intuitive\n- The paper is well written and very clear.\n- The authors clearly discuss limitations, show failure cases and provide ablations\n", "weaknesses": "- The whole work is purely constructive (not many novel conceptual insights) and most parts of the presented method are existing concepts: The method combines NeRF (as TiNeuVox), PointNeRF (adding rotation equivariance), MAT, and ARAP. However, there is novelty in the joint system. I think the positive aspects of this work outweigh this.\n- The method in general is pretty close to some of the methods for animatable human radiance fields [80-93]. It differs mostly in that it does not rely on a human template but extracts one from scratch and thus, also works on other subjects. The paper would profit from a comparison with one of them on a human subject though.\n- In general, while relevant related work is mentioned, it is not always compared against. Some relevant examples are NeRFies [6] and NSFF [4], which both are known for producing better results than D-NeRF, which is used in the comparisons.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback! We answer your specific questions in this document. Please refer to the shared response for the experiment results and for answers to common questions.\n\n**Human Subject Comparison**\n\nWe added additional results for our method applied to camera captured human full-body sequences from the ZJU-Mocap dataset [0]. Please refer to the shared response for details and results.\n\n**Further Comparisons**\n\nThank you for the suggestions, we will extend our tables with additional results of related papers that were previously benchmarked on the Blender dataset, such as Nerfies.\n\n**TiNeuVox Features**\n\nThe TiNeuVox features are used as an initialization and then further optimized during the learning process. We added additional ablations in the rebuttal PDF showing that, indeed, fine-tuning of TiNeuVox feature points is only necessary if they are randomly initialized. We optimize the color regressor ($\\Phi_c$), density regressor ($\\Phi_d$), pose regressor ($\\Phi_r$), as well as the point features ($\\mathbf{f}_i$), the point raw blend skinning vectors ($\\mathbf{\\hat{w}}_i$), and global scaling parameter $\\alpha$. We will clarify this in the manuscript.\n\n**References**\n\n[0] Peng, Sida, et al. \"Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.\" Proceedings of the IEEE/CVF CVPR 2021.\n"}]

Author rebuttal:
We thank all reviewers for their valuable feedback, and we will use it to further improve our manuscript. We are glad that all reviewers found the description of our method clear, and that they appreciate that our method does not rely on any pre-defined skeleton (RMHs), offers better novel view synthesis with training time shorter than comparable methods (ZYjY), yields good and intuitive decomposition of object parts (u7Dh) and produces compelling reposing results (bAZS).

We made our best effort to answer the reviewer's questions and we conducted as many additional experiments analyzing the behavior of our method as possible within the limited time window of the rebuttal. 

In this shared document we answer the most common points and present our additional experiments which are included in the attached PDF.

**Performance in challenging datasets and with human bodies**

We conducted an additional experiment with the ZJU-Mocap human body motion capture dataset [0]. This is a challenging dataset because it contains longer realistic motion sequences captured using cameras and human actors rather than synthetic renderings. Consequently, it also features a realistic appearance, capturing noise and complex lighting. We train our method on 5 sequences and we use 12 camera views for supervision. Each sequence consists of 490-790 frames, and we train for 320k iterations with a mask weight $w_1 = 0.2$. 

The results are presented in the attached PDF in Figure 3. We observe that our method can recover the 3D shape as well as meaningful LBS model and pose animation. However, we do recognize that the image fidelity is perceptually lower than in the case of the synthetic datasets. We attribute this partially to the necessity to model a more complex appearance and partially to known inconsistencies in the dataset described by previous work (see the discussion of imperfect camera poses and inconsistent lighting in the Supplement F of [2]).

**Impact of the Backbone Choice**

While we used TiNeuVox as the backbone in all experiments in the paper, our method is designed to be agnostic to its design. In principle, any dynamic or even static 3D shape reconstruction method could be used to initialize our approach. While we can leverage pre-trained features and network weights obtained from the backbone, our method works well even if only a coarse feature-free point cloud is available. Therefore, we opted to test this as a more challenging and general question in favor of testing any specific backbone alternative.
To this goal, we follow the same training procedure for the Robots dataset as in the main paper but we only initialize positions of our feature points ($\mathbf{p}_i^c$) while keeping their feature values ($\mathbf{f}_i$) as well as the density ($\Phi_d$) and color regressors ($\Phi_c$) random before the start of our training.

We report quantitative results in the supplement PDF Table 1. We observe that the performance is comparable to the full initialization which suggests that any backbone that defines partitioning of the scene volume is an effective initialization for our method. Additionally, we found that increasing the weight of $\mathcal{L}_\textrm{skel}$ is beneficial for performance under these conditions. We will report these findings in the final version.

**Role of Feature Point Tuning**

In our method, we jointly train multiple components of our model in an end-to-end fashion. We experimentally assessed the contribution of fine-tuning the point feature values $\mathbf{f}_i$ in Table 2 of the supplement and we found that it does not have a major impact in the Blender dataset. However, it is important to note that this only holds because the feature values were initialized from an already trained feature decoding model of TiNeuVox. Fine-tuning features is necessary if they are initialized randomly (when using an alternative backbone), as demonstrated in the experiment above. Therefore, we keep the feature fine-tuning as a part of our method for generality, and we will clarify its role in the manuscript.

**Progressive Initialization**

While our previous experiments demonstrated that our method works well with a generic baseline, we conducted an additional experiment to assess how the quality and computational budget of the baseline training affect the performance of our downstream method. Specifically, we measure the effect of the TiNeuVox-backbone pre-train iterations on the initialization of our method in the Jumping-Jacks scene from the Blender data set.

We present the results in Figure 1 of the supplement. Surprisingly, after only 100 iterations of the backbone training, our method already achieves a PSNR score of over 30 dB.  This shows that even though such early initialization is very coarse, our method still recovers fine details. However, we acknowledge that our method would not be able to recover functional object parts completely missed during the skeleton extraction phase. Therefore, we opted to conservatively train the backbone for 20k iterations in all our results.

**References**

[0]: Peng, Sida, et al. "Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans." Proceedings of the IEEE/CVF CVPR 2021.

[1]: Noguchi, Atsuhiro, et al. "Watch it move: Unsupervised discovery of 3D joints for re-posing of articulated objects." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Li, Ruilong, et al. "Tava: Template-free animatable volumetric actors." ECCV 2022.


Meta-review: This paper presents a data-driven method to discover an articulation model from a pre-trained dynamic nerf with the aid of efficient point-based rendering and LBS. It leads to It received mixed reviews. While most of reviewers found the idea of skeleton discovery interesting, there were concerned about 1) its generalization to different backbone dynamic nerfs other than TiNeuVox, 2) lack of results on real humans , 3) lack of comparisons with other D-nerfs. The authors presented additional results in the rebuttal to address these concerns. In particular, It reported experimental results on ZJU-Mocap with learned skinning weight visualization and compared with SOTA dynamic nerfs like Hexplane and K-plane. These results provided a more complete view of this work and helped addressed some of the raised concerns. As a result, the novel idea and solid design overweights the concerns. The AC would recommend its acceptance. Meanwhile, the AC urges the authors to incorporate the new results to the revision.
Decision: Accept (poster)