SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: Quality of unseen view synthesis during training with  $95\%$  confidence intervals in the Robots dataset [20]. The initial plateau of WIM [20] matches the 10k warm-up steps used by the authors before training with all data. Our onset time corresponds to the 70 minutes required for pre-training of the backbone. Training of our method was terminated after 2.5 hours.
Time	Method	PSNR↑	SSIM↑	LPIPS↓
2 h	WIM [20]	25.05	0.952	0.059
Ours	28.81	0.968	0.047
10 h	WIM [20]	27.61	0.964	0.046
Ours	30.19	0.973	0.041
Method	Reposable	PSNR↑	SSIM↑	LPIPS↓	Training Time
D-NeRF [3]	No	30.50	0.95	0.07	20 hours
TiNeuVox-B [11]	No	32.67	0.97	0.04	28 mins
Hexplane [72]	No	31.04	0.97	0.04	11.5 mins
K-Plane hybrid [71]	No	31.61	0.97	-	52min
WIM* [20]	Yes	23.81	0.91	0.10	11 hours
Ours*	Yes	29.10	0.94	0.06	2.5 hours
Table 2: Quality of unseen view synthesis for the Blender dataset [3]. Results of D-NeRF and TiNeuVox-B reprinted from Fang et al. [11]. *Without the "Bouncing balls" scene.
Figure 5: Effect of the backbone initialization pre-training steps on the final result of our method when trained on the Jumping Jacks scene from the Blender dataset. Our final choice of  $20\mathrm{k}$  iterations corresponds to approximately 70 minutes of real time.
Fig. 6 right). We use the original training-test split. Third, ZJU-MoCap dataset [82] is a common test suite for dynamic human reconstruction and we evaluate 5 of its multi-view sequences with the same 6 training views as used in Watch-It-Move [20]. Finally, we evaluate image quality using peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [114], and learned perceptual image patch similarity (LPIPS) [115] image metrics.
Implementation details We pre-train the TiNeuVox [11] backbone using the authors' implementation and an additional distortion loss [116], as implemented in [117]. Our method is implemented in Pytorch and we train each scene using the Adam optimizer for 160k (Blender and Robots) or 320k (ZJU-MoCap) iterations with a batch size of 8192 rays, sampled randomly from multiple views and a single timestamp. We choose the canonical timestamp by visual inspection, and gradually increase the number of observed timestamps during training. We adjust ray sampling and scheduling for the ZJU-MoCap dataset (see the Supplement). All experiments were done on a single Nvidia GPU RTX 3090Ti. See the supplementary materials for details. For more details, see the Appendix.
Baselines We compare our method to state-of-the-art non-articulated and articulated methods for high-fidelity multi-view video synthesis (see Table 1 for an overview).  $D$ -NeRF [3] extends NeRF to the temporal domain by backward warping a static canonical NeRF. TiNeuVox [11] improves performance of  $D$ -NeRF using voxel grids. Hexplane [72] and  $K$ -Planes [71] decompose to the space-time volume to several hyper-planes. Finally, Watch-It-Move [20] (WIM) jointly learns a surface representation and LBS model for articulation. Note, that because we aim at time-efficient
Figure 6: While our model well reproduces details and non-rigidity of the *Dinosaur* (right), it can fail for complex motions combining long chains of rotations with texture cue ambiguity (left).
Figure 7: Learned LBS weights and skeleton: a) After training. b) After additional post-processing (weight merging and skeleton pruning, see Sec. 4.2).
learning, training of WIM was stopped after 11 hours (i.e.,  $80\mathrm{k}$  of the original  $400\mathrm{k}$  optimization steps).
Novel view synthesis We provide results for the Robots view synthesis without the skeleton simplification; quantitatively in Fig. 4 and qualitatively in Fig. 3. More visual examples are available in the supplement. After the initial pre-training, our method quickly surpasses the image quality of WIM [20]. The mean image scores obtained for our method after 2 hours of training (incl. pretraining) are higher than those that WIM achieves after 10 hours. We attribute this to NeRF's capability to approximate even complex shapes, which are difficult to fit using the signed distance function utilized by WIM. Nevertheless, for visually simple objects with long nested pose transformation chains, this capacity might encourage false explanations of the articulation and cause artifacts, as visible in Fig. 6 left. However, this is not a common issue. We illustrate it in the Blender, where WIM struggles to represent fine details (see Fig. 6 right), while our method achieves image scores close to those of non-reposable baselines (see Table 2 and the supplement).
Furthermore, Fig. 5 shows that our image quality is not highly dependent on the pre-training phase. We observe that mere 100 iterations of TiNeuVox pre-training provide an initialization sufficient for achieving PSNR scores over  $30\mathrm{dB}$  by our method. However, while fine geometric details are still recovered, the coarse initial shape limits the skeleton complexity and, therefore, we opt for  $20\mathrm{k}$  pre-training steps in all other experiments.
ZJU-MoCap In Fig. 10, we compare our method to WIM in the ZJU-MoCap dataset. We observe that both methods are able to recover the 3D shape and the skeletal motion despite the difficulty of an accurate fine texture detail reconstruction. This can partly be attributed to imperfections in camera calibrations (see Supplement F of [96]) and partly by soft deformations of clothes which are modeled neither method. Notably, with a small modification our method learns to partially compensate for this. In Ours $^{pose}$ , we condition the feature embedding network  $\Phi_p$  by the skeletal poses jointly learned from scratch by our method. This improves the image quality by modeling the residual deformations. See the Supplement for details. Finally, Ours $^{SMPL}$  shows that replacing our skeleton in Ours $^{pose}$  with a ground truth from an annotated SMPL template [15] does not affect the performance. This validates our skeleton initialization based on Medial Axis Transform.
Figure 8: Reposing using the simplified skeleton. Interpolation from canonical to novel pose.
Figure 9: Ablation examples. a)  $\mathcal{L}_{ARAP}$  enforces rigidity after pruning (see upper pipes), b)  $\mathcal{L}_{smooth}$  results in better part-segmentation, c)  $\mathcal{L}_{skel}$  enforces the joints to stay inside the shape.
Figure 10: Comparison in the ZJU-MoCap dataset. Ours: Our full method.  $\text{Ours}^{pose}$ : Ours with an additional pose-conditioned feature embedding network  $\Phi_p$ .  $\text{Ours}^{SMPL}$ :  $\text{Ours}^{pose}$  with ground truth SMPL [15] skeleton.
Method	PSNR↑	SSIM↑	LPIPS↓
WIM [20]	31.08	0.963	0.053
Ours	29.60	0.958	0.063
Ourspose	32.05	0.967	0.056
OursSMPL	32.01	0.967	0.056
Reposing In Fig. 7, we visualize the learned blend-skinning weights  $\hat{\mathbf{w}}_i$  and skeletons with and without the skeleton simplification. The algorithm is able to substantially reduce the skeleton complexity, while largely preserving semantic articulations observed in the training data. However, it is not able to remove all unnecessary skeletal branches when complex geometry is present (see Fig. 7 right). In Fig. 8, we show that such post-processed skeletons allow for animating of novel poses by smoothly interpolating between user-defined poses. More animations can be found in the Supplement.
Loss ablation Our experiments suggest the regularization does not improve the image quality, but it improves the quality of the kinematic model, which is important for our main goal of reposing (see Fig. 9). Specifically,  $\mathcal{L}_{ARAP}$  avoids non-rigid distortions (a),  $\mathcal{L}_{smooth}$  leads to a more semantically meaningful part segmentation (b),  $\mathcal{L}_{skel}$  encourages functional placement of the skeleton joints inside the object volume even for thin parts (c), and  $\mathcal{L}_{tranf}$  leads to a reduction of necessary object parts after simplification (62 components are removed instead of 49 for Atlas). More details are provided in the Supplement.
Backbone The TiNeuVox backbone allows us to transfer parameters learned during the pre-training and initialize the features  $\mathbf{f}_i$  and the regressors  $\Phi_d$  and  $\Phi_c$ . Interestingly, our experiment shows that such off-the-shelf features  $\mathbf{f}_i$  often do not need any further fine-tuning (see Table 4). Despite this, our method is agnostic to the backbone choice by design and the end-to-end training procedure of the entire model is essential for this. We demonstrate it by training with a random initialization. This way, only the point positions  $\mathbf{p}_i^c$  obtainable by any 3D reconstruction method are needed. Table 3 shows that this leads to only a negligible drop in reconstruction quality in the Robots dataset. Here, the weight of  $\mathcal{L}_{skel}(w_2 = 1)$  was adjusted to prevent drift of the skeleton.
PSNR↑	SSIM↑	LPIPS↓
Full Method	30.19	0.973	0.041
Random Init.	29.97	0.971	0.045
Table 3: Results with a TiNeuVox initialization and with a random initialization of the features  $\mathbf{f}_i$  and the regressors  $\Phi_d$  and  $\Phi_c$  in the Robots dataset.
PSNR↑	SSIM↑	LPIPS↓
Full Method	29.10	0.94	0.06
No Fine-tuning	29.16	0.94	0.05
Table 4: Results with and without fine-tuning of the feature points  $\mathbf{f}_i$  in the Blender dataset.
Limitations and Future Work We demonstrate fast learning of articulated NeRFs for state-of-the-art view synthesis and straightforward skeletal reposing for objects with piece-wise rigid pose transformations. While the LBS allows for fitting partially non-rigid deformations (see Fig. 6 right), representing fully non-rigid, topologically varying, and multi-component objects would benefit from a higher-dimensional parameterization. Chen et al. [102] aim in that direction, but an intuitive and cost-effective reposing still favors skeleton-based techniques such as ours.
The performance of our method depends on the quality of the backbone reconstruction. While TiNeuVox [11] supports reconstruction from even sparse data, a different backbone could offer a more robust starting geometry and improve the skeleton initialization. Although our approach works well even for highly structured shapes and thin structures (see Fig. 6 right), we observe incorrect joint placement for objects with long chains of highly nonlinear geometrical transformations (see Fig. 6 left). Moreover, our approach successfully reduces the number of extraneous bones but it is not able to completely eliminate all superfluous skeletal elements (see Fig. 7).
Our method is restricted to the kinematic motion space exhibited in the training sequences. While extrapolation is possible to some extent, the proposed method is not able to generalize to arbitrary unseen poses. Finally, we focus on image synthesis for user-defined poses rather than a direct fitting of unseen skeletal poses from images or motion capture. An inverse fitting of skeletal poses to novel 2D observations or transfer of poses from one object to another remain opportunities for future research.
Conclusion We presented a method for fast learning of articulated models for high-quality novel view synthesis of captured 3D objects. Our forward-warped neural point clouds avoid the pitfalls of backward warping and elegantly integrate a skeleton-based LBS. As a result, our method merges straightforward reposing even of strongly animated inputs, as present in the Robots dataset, with high visual fidelity of NeRF rendering. Our work is a significant step towards low-cost acquisition of animatable 3D objects for games, movies, and education.
Ethical Considerations Our method renders novel views and poses of 3D objects including human bodies. However, we do not focus on this class and we note that many human-specific methods exist (see Sec. 2). Nevertheless, we acknowledge that our method could potentially be used to produce fake images of people and that research is needed to understand the risks and their mitigation.
[116] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
[117] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Improved direct voxel grid optimization for radiance fields reconstruction. arXiv preprint arXiv:2206.05085, 2022.