SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. The tolerance to SMPL inaccuracy varies among different methods. In the GeneBody dataset, the provided masks and SMPL parameters are not as accurate as ZJU-MoCap. So, we can see from the columns of GeneBody Normal even though the human actors do not wear complex clothes or perform hard poses, NeuralBody still performs poorly due to inaccurate SMPL parameters. The reason behind this has been discussed in Sec. 3.4.
2. Existing scene-specific methods still perform poorly in challenging cases. As demonstrated in the columns GeneBody Hard Cloth and GeneBody Hard Pose, both NeuralBody and HumanNeRF have a severe performance drop on all metrics compared to the results on ZJU-MoCap. The challenges brought by hard clothes are two folds. The first is that the motion of clothes (especially loose clothes) cannot be depicted by SMPL model. The other one is that complex clothes have more texture details and geometry variations. They lead to increased difficulties for both appearance and body deformation modeling in the sense of more data inconsistency. As for hard poses, the challenge is the larger variations of body and the corresponding cloth deformation, which increases the difficulties for body deformation modeling.
3. Compared to hard poses, hard clothes are more challenging for existing models. Comparing the column GeneBody Hard Cloth and GeneBody Hard Pose, it can be observed that hard clothes have a more severe impact on the overall performance for all representative methods than hard poses. This is because hard clothes affect both appearance and body deformation modeling, while hard poses mainly affect body deformation modeling. The qualitative results are shown in Fig. 4
4. Existing scene-specific methods have a consistent performance drop for novel pose rendering. Comparing the results in the top rows (novel view rendering) and bottom rows (novel pose rendering), we can find that with the same settings and training data, the selected representative methods have
inferior performance for novel pose rendering compared to novel view rendering. This indicates that the learned body deformation cannot generalize well enough to unseen poses for existing scene-specific methods, especially significantly increasing the LPIPS.
Table 7: Extensive quantitative comparison of representative scene-specific methods on ZJU-MoCap and different partitions of GeneBody.
Methods	Novel view rendering
ZJU-MoCap	GeneBody normal	GeneBody hard cloth	GeneBody hard pose
PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓
NeuralBody [38]	27.82	0.9396	0.1048	19.31	0.8499	0.3197	16.24	0.7748	0.3093	19.99	0.8412	0.2467
HumanNeRF [49]	25.86	0.9250	0.0728	24.63	0.8865	0.2319	17.36	0.7729	0.2155	21.56	0.8388	0.1715
Methods	Novel pose rendering
ZJU-MoCap	GeneBody normal	GeneBody hard cloth	GeneBody hard pose
PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓
NeuralBody [38]	23.73	0.8871	0.1525	18.79	0.8325	0.3402	14.47	0.7039	0.3657	17.21	0.7801	0.2985
HumanNeRF [49]	23.64	0.8907	0.1027	24.61	0.8861	0.2254	15.60	0.7331	0.2610	19.07	0.7946	0.2122
4.2 Benchmarking Generalizable Methods on HuMMan
For generalizable methods, the generalization mainly refers to novel view rendering for unseen human actors after pre-training on the multi-view images of a certain number of human actors. The scale and diversity of training data are of great importance to achieve good generalization. To study the potential of this stream of methods, We evaluate the in-domain (i.e., cross-subject) and out-of-domain (i.e., cross-dataset) generalization ability the selected representative generalizable method, i.e. GP-NeRF.
From Tab. 8, the test results consistently perform better than the model trained on the indomain dataset. Even using a large-scale HuM-Man dataset to train the model and cross-dataset test, the performance is still worse than the model trained on ZJU-MoCap. This indicates that the existing generalizable methods still cannot achieve satisfying performance when rendering novel views for an unseen human in real scenarios. The state-of-the-art generalizable
method GP-NeRF tends to overfit small-scale data due to the higher performance on in-domain ZJU-MoCap training and testing.
Table 8: Quantitative comparison of cross-subjects generalization of GP-NeRF. ZJU-MoCap 7 and ZJU-MoCap 3 mean the train and test split of ZJU-MoCap, respectively.
Train set	Test set	PSNR ↑	SSIM ↑	LPIPS ↓
ZJU-MoCap 7	ZJU-MoCap 3	26.66	0.9263	0.1256
HuMMan	ZJU-MoCap 3	23.82	0.8810	0.1850
ZJU-MoCap 7	HuMMan Eval	17.89	0.8833	0.2318
HuMMan	HuMMan Eval	21.68	0.9234	0.1511
Upper-bound performance of finetuning on a single subject. For generalizable methods, to further increase the performance of novel view rendering for an unseen human, one effective way is to finetune the trained model on the human's multi-view sequences. To test the upper-bound performance of finetuning, we use the pretrained model of GP-NeRF to train three subjects of ZJU-MoCap separately for a long enough time and record the quantitative performance after 15 minutes, 1 hour, and 20 hours, respectively. As shown in Fig. 5 and Tab.9, benefiting from pretraining, GP-NeRF has higher PSNR than HumanNeRF from the early start of finetune but increases slowly even after a long training time. In contrast, the scene-specific method HumanNeRF is trained from scratch with the same settings, and it performs poorly initially, but its PSNR grows faster than GP-NeRF. After enough time for finetune (training), HumanNeRF has significantly lower LPIPS so
Figure 5: Rendering quality grows with the increase of finetuning time. The generalizable method GP-NeRF initially looks better than the scene-specific method HumanNeRF, but Human-NeRF has visually better results after several hours.
that the rendered images by HumanNeRF look realistic and high-quality to the GT images.
Table 9: Quantitative comparison of the upper bound performance of finetuning on a single subject using a pretrained model for the generalizable method GP-NeRF. For reference, we also train HumanNeRF from scratch given the same train views, frames, and time.
Methods	15 mins finetune	1 hour finetune	20 hours finetune
PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓	PSNR ↑	SSIM ↑	LPIPS ↓
GP-NeRF [8]	27.46	0.9275	0.1464	27.94	0.9330	0.1390	28.80	0.9408	0.1290
HumanNeRF [49]	22.22	0.8549	0.1772	23.26	0.8779	0.1285	25.36	0.9156	0.0807
4.3 Exploring the Key to Animatability
Scene-specific methods. For scene-specific methods, the definition of animatability is equivalent to novel pose rendering, i.e., rendering a human in a given pose. Thus, as discussed in Sec.4.1, existing scene-specific methods are all animatable, but the rendering quality would be worse than novel view rendering. The key element of animatability for scene-specific methods is a module to model the body deformation so that they can use images of a human performing different actions for training. For example, NeuralBody encodes the appearance information of a human into SMPL vertices, so it can directly use the body deformation module of SMPL to animate a human body into any given pose. In contrast, HumanNeRF encodes appearance information of a human of T-pose in canonical space and decouples the body deformation into skeletal motion, which is modeled by an implicitly learned LBS (linear blend skinning) weight field, and non-rigid motion, which is modeled by an implicitly learned translation field. Most of the existing scene-specific methods follow these two paradigms. From our experiments in Sec. 3.4, the latter has more tolerance to inaccurate SMPL parameters, while the former is easier to implement.
GeneHumanNeRF: making HumanNeRF generalizable. Existing generalizable methods are not animatable, except for MPSNeRF [16], because they require a certain number of views as input images to render novel views. For generalizable methods, the appearance information of a human to be rendered is acquired from input images. Thus, these methods cannot directly work by merely specifying a pose without reference to multi-view input images.
Table 10: The performance of our built simple baseline GeneHumanNeRF for animatable and generalizable human body rendering from a monocular video.
Train set	Test set	PSNR ↑	SSIM ↑	LPIPS ↓
ZJU-MoCap 7	ZJU-MoCap 3	22.87	0.8647	0.2110
HuMMan	ZJU-MoCap 3	22.73	0.8573	0.2290
ZJU-MoCap 7	HuMMan Eval	18.09	0.8715	0.2159
HuMMan	HuMMan Eval	21.06	0.8882	0.2010
MPSNeRF achieves animatability by exploiting LBS warping twice so that the appearance information of a posed human in the input multi-view images can be warped to the rendered human in given poses. The key element is a canonical pose space where the appearance information from input images is first warped to and from which the appearance information is warped to the target pose space. We find that such an element is not limited to the setting that requires multi-view images as input. Therefore, borrowing this effective module, we built a new generalizable and animatable method that takes a few monocular video frames as input (e.g., [16]), which is like an extension of HumanNeRF from the setting of per-scene trained into one-shot trained. We name this new baseline method as GeneHumanNeRF. The motivation for this baseline method is that monocular videos are easier to acquire, so it is more suitable for practical applications. The more technical details of this baseline can be found in supplementary material. We build the benchmark on both ZJU-MoCap and HuMMan datasets and evaluate their cross-subject and cross-dataset generalization ability in Tab. 10. The performance of the proposed simple baseline can achieve a competitive performance on both datasets, which can be a good reference for future works.
5 Inspirations for Future Works
5.1 Unifying Settings
As we can see from the tables presented in Sec. 3, chaotic train & evaluation settings make the community hard to distinguish the advantages and drawbacks of existing methods from their paper-reported results. And our ablation studies for different settings indicate that those settings can make a difference. Therefore, for future works, it is highly recommended to conduct experiments in the
settings consistent with those used in this benchmark and report their results of more settings (more datasets, different frame & view settings and so on). In addition, most works assume accurate SMPL parameters are available by default, but it is not true especially for real applications. Thus, how to reduce the adverse impact of inaccurate SMPL parameters can also be a problem worthy of studying.
5.2 Scene-specific Methods
For scene-specific methods, future works should focus more on the scenes that people wearing complex clothes and improve the pose generalization. To evaluate the real performance on these aspects, more datasets should be included for experiments. Besides, a proper sampling strategy for train frames and views can also lead to a significant performance increase. As most existing methods rely on SMPL to capture the human motions, it is worth thinking is there any alternative model that can depict the human geometry more precisely and whose body parts move more naturally.
5.3 Generalizable Methods
For generalizable methods, future works should focus on the improvement of cross-dataset generalization. Currently, existing methods train on small-size datasets and evaluate on the in-domain test sets. Although the quantitative results look good, they have poor performance when tested on other datasets that have more diverse cases. Due to the high cost for acquiring multi-view video sequences, data scarcity may also be an obstacle to training a generalizable model that can perform well in real scenarios. On the other hand, animatability is not compatible with the common settings of existing generalizable methods, but animatability is expected for many applications. Therefore, building a high-performance animatable and generalizable model that takes a monocular video as input can lead to interesting applications. We have proposed a simple baseline of this setting to be a reference for future works.
In this work, we build the inaugural comprehensive benchmark for neural human radiance fields. Specifically, we unify the evaluation settings and metrics. We introduce more challenging datasets and establish the benchmark for them. To explore the capability of existing generalizable models, we train them on large-scale datasets and conduct cross-subject validation. Lastly, after analyzing the key components of animatability or generalizability, we design a baseline that caters to both these attributes given monocular videos. We sincerely hope these efforts can benefit this field of study.
Acknowledgement The work was supported in part by NSFC with Grant No. 62293482, the Basic Research Project No. HZQB-KCZYZ-2021067 of Hetao ShenzhenHK S&T Cooperation Zone, the National Key R&D Program of China with grant No. 2018YFB1800800, by Shenzhen Outstanding Talents Training Fund 202002, by Guangdong Research Projects No. 2017ZT07X152 and No. 2019CX01X104, by the Guangdong Provincial Key Laboratory of Future Networks of Intelligence (Grant No. 2022B1212010001), and by Shenzhen Key Laboratory of Big Data and Artificial Intelligence (Grant No. ZDSYS201707251409055). It was also partially supported by NSFC62172348, Outstanding Young Fund of Guangdong Province with No. 2023B1515020055 and Shenzhen General Project with No. JCYJ20220530143604010. In addition, we really thank Yuqi Hu from HKUST (GZ) for helping code cleaning and build the project webpage.
[46] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2021.
[47] Shaofei Wang, Katja Schwarz, Andreas Geiger, and Siyu Tang. Arah: Animatable volume rendering of articulated human sdfs. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII, pages 1-19. Springer, 2022.
[48] Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. Hf-neus: Improved surface reconstruction using high-frequency details. Advances in Neural Information Processing Systems, 35:1966-1978, 2022.
[49] Chung-Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher-Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16210-16220, 2022.
[50] Zhiwen Yan, Chen Li, and Gim Hee Lee. Nerf-ds: Neural radiance fields for dynamic specular objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8285-8295, 2023.
[51] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4815, 2021.
[52] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoptrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752-5761, 2021.
[53] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2021.
[54] Zhengming Yu, Wei Cheng, Xian Liu, Wayne Wu, and Kwan-Yee Lin. Monohuman: Animatable human neural field from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16943-16953, 2023.
[55] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[56] Yongqiang Zhang, Zhipeng Hu, Haoqian Wu, Minda Zhao, Lincheng Li, Zhengxia Zou, and Changjie Fan. Towards unbiased volume rendering of neural implicit surfaces with geometry priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4359-4368, 2023.
[57] Zerong Zheng, Tao Yu, Yixuan Wei, Qionghai Dai, and Yebin Liu. Deephuman: 3d human reconstruction from a single image. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
[58] Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12588-12597, 2023.