SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Methods for RenderMe-360 benchmarks. M: Multi-view images.; S: Single-view images, C: Camera Calibration, V: Neural Volumetric, S: Neural SDF, P: Point-based Representation, F: Feature Space, S: Static, D: Dynamic, I: Images Conditioning, L: Latent Codes, P: Parametric Models K: Face Keypoints, R: Radiance Field-based, F: SDF-based, N: Convolution-based, S: Case specific, G: Generalizable. Better zoom in.
Attribute	Novel View Synthesis	Novel Expression Synthesis	Hair Rendering	Hair Editing	Talking Head
Required Data	M+C	M+C	M+C	M+C	M+C	M+C	M+C	S+C	S+C	S+C	M+C	M+C	S	S	S+C	S+C
Representation	V	S	V	V	V	V	V	V	V	P	V	V	F	F	V	V
Static/Dynamic	S	S	D	D	S	S	S	D	D	D	D	D	S	S	D	D
Conditioning	I	I	I	I	I	I	I	L	L	L	I	I	L	L	L	L
Face Priors	X	X	X	P	X	K	X	P	P	P	X	X	F	F	P	P
3D Consistency	R	F	F	R	R	R	R	R	R	R	R	R	N	N	R	R
Generalizability	S	S	S	S	G	G	G	S	S	S	S	S	G	G	S	S
Settings. We select 20 identities from the three categories to evaluate the methods. For the training-testing split, we uniformly sample 22 views from all 60 views as test views, and use the rest camera views to train each model. A visualization of camera distribution is shown in Supplementary Materials,
noted as  $Cam1$ . The four methods for comparison are: Instant-NGP [34], NeuS [48], NV [30] and MVP [31]. The first three methods are originally designed for general-purpose case-specific NVS, and the last one is designed for human head avatar reconstruction. We compute PSNR, SSIM, and LPIPS for rendered novel view images against ground-truth. The quantitative results are listed in Table 3, and the qualitative ones are listed in Supplementary Materials.
Results. We observed three key phenomena under #Protocol-1: 1) All methods tend to drop the performance lengthways along the Table 3, with the level of accessory complexity increasing. This phenomenon reflects the status quo that we do not yet have one strong paradigm for robust case-specific human head multi-view reconstruction. 2) NeuS yields the best performance on average. There are two possible underlying reasons. First, NV and MVP are dynamic methods while not emphasizing temporal-consistency constraints. Thus, when comparing these methods with static ones under static measurement, the perturbation of data sequences would affect these two methods' construction on dynamic fields to certain degrees. Second, by associating the quantitative results with the qualitative ones, we can find that NeuS performs well in global shape reconstruction with almost no surrounding noise due to its surface representation property, but has a much smoothing surface appearance. In contrast, Instant-NGP and MVP can recover better high-frequency details. MVP uses multiple-primitive representation with different networks to render, equipping the model with a larger representative capacity. Whereas, they produce more surrounding noise. Neural Volume renders images mostly with artifacts. We could draw the idea that surface representation helps the novel view reconstruction in a global shape-forming manner. 3) NV suffers from limited grid resolution (although it uses inverse warping to ease the problem) and inaccurate alpha value estimation. Thus, it introduces more artifacts than others, and is strenuous in reconstructing high-frequency details.
Table 3: Single ID NVS (#Protocol-1). We evaluate methods under three subsets with levels of complexity.
Split	Metrics	NGP [34]	NeuS [48]	NV [30]	MVP [31]	Split	Metrics	NGP [34]	NeuS [48]	NV [30]	MVP [31]
Normal Case	PSNR↑	24.71	26.29	19.61	23.65	With Complex
Accessory	PSNR↑	20.54	22.89	16.46	21.5
SSIM↑	0.848	0.927	0.777	0.895	SSIM↑	0.776	0.874	0.598	0.83
LPIPS↓	0.28	0.11	0.29	0.14	LPIPS↓	0.36	0.16	0.44	0.18
With Deformable
Accessory	PSNR↑	23.06	23.53	17.83	23.93	Overall	PSNR↑	23.21	24.67	18.56	23.1
SSIM↑	0.807	0.904	0.703	0.893	SSIM↑	0.819	0.906	0.723	0.876
LPIPS↓	0.31	0.13	0.34	0.12	LPIPS↓	0.31	0.13	0.33	0.15
4.1.2 Generalizable NVS
This track refers to the setting of training across multiple human heads, and testing on unseen $^{1}$  human heads (i.e., new identities) or unseen motions (e.g., expressions) with conditioning on one or few input images (as source views). It allows us to evaluate the network's effectiveness in learning priors, and the ability to adapt priors. We investigate three methods under two protocols in the main paper - 1) #Protocol-1 for investigating methods' generalization ability on geometry deformation via evaluating the generalization ability to unseen expressions on seen identities. 2) #Protocol-2 for probing methods' capability in learning category-level human head priors via evaluating the generalization ability to unseen identities. We also name this protocol as Unseen ID NVS; This setting is challenging as it requires the model to generalize to both new appearances and geometries. To further reveal the factors that might have influences on generalization, we enrich both protocols with four sets of training-testing view settings and three data subsets under different complexity.
Settings. We study three generalizable methods: IBRNet [49], VisionNeRF [24], and Keypoint-NeRF [32]. For training-testing identity split, we select a subset from RenderMe-360, with 160 identities for training and 20 for serving as unseen identities. The selected identities are evenly sampled from the three data subsets. We select 7 out of 60 camera views as novel views. Note that, we calculate the metrics in #Protocol-2 on all expressions. As a consequence, 10 expression structures attached with trained identities are covered in training set, and the rest 2 expression structures are unseen. Such an evaluation strategy provides the feasibility for researchers to analyze their methods' generalization ability on appearance and geometry in both entangled and disentangled aspects.
Results. The quantitative results are shown in Table 4. From per method perspective, we draw the consistent conclusions that: 1) random view training could help enhance the model's robustness on both unseen expression and identity tasks; 2) the performance declines in terms of most metrics with the complexity of human head's appearance/geometry increase; 3) the Unseen ID NVS task introduces larger performance drop rate than Unseen Expression NVS. These two phenomena suggest that these generalizable methods could learn priors like the information of 'minimal-accessory'
<sup>1</sup>Note that the adjective 'seen' refers to the sample that is used in training, and 'unseen' means the sample is not used as training data.
Table 4: Benchmark results on generalizable NVS. The results are evaluated on: (1) unseen expressions of sampled training identities, and (2) unseen identities in each test split. (LPIPS* denotes LPIPS × 1000)
Training Setting	Testing Setting	Methods	Normal Case	With Deformable Accessories	With Complex Accessories	Overall
PSNR†	SSIM†	LPIPS↓*	PSNR†	SSIM†	LPIPS↓*	PSNR†	SSIM†	LPIPS↓*	PSNR†	SSIM†
Fixed Views	IBRNet[49]	23.36	0.918	144.17	20.82	0.849	197.85	20.33	0.827	187.57	21.97	0.878	168.44
VisionNeRF[24]	23.57	0.905	139.52	20.42	0.846	186.04	20.89	0.835	189.60	22.11	0.873	163.67
KeypointNeRF[32]	19.59	0.898	127.06	17.42	0.805	213.43	16.54	0.760	205.82	18.29	0.840	168.34
Random Views	IBRNet[49]	24.34	0.924	140.21	20.81	0.85	20.45	0.832	179.57	22.485	0.883	162.39
VisionNeRF[24]	25.79	0.914	148.70	21.43	0.883	148.90	20.37	0.87	23.345	0.895	151.45
KeypointNeRF[32]	16.96	0.871	170.66	16.07	0.775	256.83	14.64	0.714	270.33	16.16	0.808
Unseen Expression NVS	IBRNet[49]	23.37	0.918	144.21	20.82	0.8487	197.85	19.79	0.803	182.85	21.84	0.872	167.28
VisionNeRF[24]	23.05	0.905	135.20	21.42	0.864	167.00	20.28	0.835	165.01	21.95	0.877	150.60
KeypointNeRF[32]	19.74	0.902	113.5	18.05	0.817	183.66	17.02	0.778	182.08	18.64	0.850	148.19
Random Views	IBRNet[49]	24.38	0.924	139.71	21.02	0.850	190	20.91	0.837	175.14	22.67	0.884
VisionNeRF[24]	28.08	0.943	97.32	23.86	0.882	150.6	23.08	0.873	133.2	25.78	0.910
KeypointNeRF[32]	18.65	0.897	124.33	17.60	0.813	192.04	16.61	0.779	186.67	17.88	0.847
Unseen ID NVS	Training Setting	Testing Setting	Normal Case	With Deformable Accessories	With Complex Accessories	Overall
PSNR†	SSIM†	LPIPS↓*	PSNR†	SSIM†	LPIPS↓*	PSNR†	SSIM†	LPIPS↓*	PSNR†	SSIM†
Fixed Views	IBRNet[49]	22.25	0.895	157.96	18.42	0.824	213.55	17.97	0.744	255.98	20.22	0.840	196.36
VisionNeRF[24]	21.01	0.866	146.44	18.00	0.801	216.22	17.35	0.734	262.60	19.34	0.817	192.93
KeypointNeRF[32]	18.85	0.866	148.13	15.93	0.789	205.04	16.14	0.734	231.89	17.44	0.814	183.30
Random Views	IBRNet[49]	22.54	0.897	154.06	18.72	0.831	198.75	18.12	0.751	249.35	20.48	0.844
VisionNeRF[24]	24.01	0.818	150.04	18.15	0.857	198.46	19.33	0.796	197.42	21.38	0.822
KeypointNeRF[32]	17.03	0.841	187.19	14.79	0.76	244.94	15.46	0.715	273.00	16.08	0.789
Random Views	IBRNet[49]	22.24	0.895	157.95	18.42	0.824	213.55	18.01	0.746	256.81	20.23	0.840	196.57
VisionNeRF[24]	21.92	0.889	139.90	18.43	0.833	176.12	18.35	0.773	223.04	20.16	0.846	169.74
KeypointNeRF[32]	18.96	0.868	138.21	16.15	0.800	185.43	16.12	0.744	230.09	17.55	0.820	172.99
Random Views	IBRNet[49]	22.53	0.897	154.05	18.75	0.830	195.12	18.10	0.749	250.72	20.48	0.843
VisionNeRF[24]	24.77	0.918	110.4	20.22	0.858	149.30	19.35	0.797	196.90	22.28	0.873
KeypointNeRF[32]	18.02	0.865	145.30	15.75	0.794	194.16	16.15	0.747	227.49	16.99	0.818
Figure 4: Qualitative Results of Generalizable NVS (#Protocol-1&2). We illustrate three generalizable methods in two different settings. The regions in red boxes are zoomed in for better visualization.
mean head, and local geometry transformation on a certain level, while still struggling with more diverse scenarios that are long-tail distributed (please associate statistical details in Supplementary Materials). In addition, there are several interesting observations when comparing the three methods: 1) VisionNeRF[24] achieves the best results on average. The robustness might come from its large capacity of learnable variables from a transformer-based structure on image features and the multi-resolution based encoder. 2) IBRNet[49] results in blurry synthesis even under the train and test settings on fixed views. 3) KeypointNeRF [32] falls behind for most of the scenarios, but is in the lead on LPIPS on average. In other words, KeypointNeRF benefits in perceptual measurement like LPIPS while suffering from pixel-wise measurements. We infer the possible reason behind the contradictory metric performances is that – the modules driven by triangulated keypoints provide better feature and view alignments in an explicit manner to help reconstruct the radiance fields. Whereas, such a key insight is a double-edged sword for full human head tasks. Since only the facial region could be well guaranteed with accessible facial landmarks. As a consequence, non-facial regions, like the hat in Figure 4, are more blurry than the facial region and distorted in the geometry aspect. Moreover, KeypointNeRF only renders the intersected frustum regions from source views in practice, which aggravates the performance problem from the full-head measurements. The results turn better when we only calculate regions that KeypointNeRF could render, as shown in Supplementary Materials.
4.2 Intra-Dataset Evaluation
Settings. To demonstrate the strengths of our dataset's diversity and scale, we first conduct an intra-dataset evaluation with different subset settings. Concretely, we separate the training data into three subsets according to accessories difficulties, namely, 'Subset 1-3' which corresponds to 'Normal Case', 'With Deformable Accessories', and 'With Complex Accessories' respectively. We also randomly sample subsets with  $30\%$  and  $50\%$  of the training data to investigate the effectiveness of the data scale. We train models (three generalizable methods [24, 49, 32]) on different subsets, and follow the strategy of #Protocol-2 in Sec. 4.1.2. To ensure fairness, we stop model training on the same global step with the same learning rate.
Results. We span the results of models trained on different training sets by test splits. The quantitative results of IBRNet [49] are visualized in Fig. 5. Please refer to supplementary for detailed numbers and qualitative results. Generally, models trained on one subset achieve relatively better performance on test split of the same category than the ones train on other subsets. For instance, the
model trained on 'Subset 1' gets the best performance in "Normal Case" splits while performing unsatisfactorily in other splits with accessories. With the full training data set, the model tends to overfit less on specific splits and achieves better overall performance. For models trained on smaller random datasets with even distributed data coverage, there is no obvious overfitting on any certain split. While we can observe that with a larger scale of data, the model generalizes better on novel identities. To sum up, we believe the proposed dataset could improve the robustness of generalization methods with both the large scale and the wide range of data distribution.
Figure 5: Intra-Dataset evaluation. We visualize quantitative results by training and testing on different subsets of the proposed dataset. Smaller bubbles indicate better performances. Better zoom in for details.
Settings. We conducted cross-dataset experiments to validate the transferability of our dataset. Specifically, we use Multiface dataset [51] for comparison. Similar to intradataset evaluation, three aforementioned generalization methods [49, 24, 32] used in Section 4.1.2, are utilized to validate the cross-domain performance of our proposed RenderMe-360 and Multiface. We rigidly transform the Multiface's camera system to align with our dataset so as to avoid coordinate offsets. For fairness, we also crop and resize both source and target images with  $512 \times 512$ , and train models with the same global step and learning rate. We also follow the strategy of #Protocol-2 in Sec. 4.1.2, models trained on both datasets are directly evaluated on test sets without any fine-tuning.
Results. We visualize the cross-dataset performance of three methods in Fig. 6, where we can observe that our dataset achieves excellent cross-domain performance in all three generalizable methods [32, 49, 24]. Compared to RenderMe-360, Multiface dataset only contains 13 identities. Thanks to the
large data coverage and scale of RenderMe-360, it enables robust generalization on novel identities in both in-domain and cross-domain settings. Especially, when testing on the Multiface test set, the cross-domain models trained on our dataset even outperform the in-domain evaluation of Multiface-trained models.
5 Boarder Impact and Limitations
Figure 6: Cross-Dataset evaluation. We visualize the quantitative results by training and testing on the different datasets. Better zoom in for details.
Previously, the unavailability of publicly accessible datasets comprising large-scale high-fidelity human heads had impeded advancements in related fields for years. The proposed RenderMe-360 dataset, together with the comprehensive benchmark, is expected to advance the research community in developing high-fidelity head avatars. It also brings new opportunities to many downstream applications in various fields, such as video telephony and the film industry, which shall be beneficial for society. However, there are still several challenges beyond the current scope of this project, that are essential to tackle in the future. We discuss the challenges from both ethical consideration and dataset limitation aspects in the following subsections.
Ethical Consideration. Both our dataset and future research efforts based on it, might be applied for malicious purposes like identity theft and fake news. For example, one might use the data to generate fake videos performing various talking interactions with realistic appearances, head/facial/lip motions, and audio. Then, the generated videos might be used in illegal activities such as financial fraud and emotional harm. This might lead to social instability and inflict emotional harm. To alleviate the negative impacts, there are three doable solutions. (1) DeepFake detection algorithms could be applied to evaluate the realism of videos. In this way, the public could be informed whether the video content is real or fake. Besides, our dataset, as well as the data synthesized by algorithms (from our benchmarks), could also serve as references for advancing the development of deepfake detection algorithms. (2) Watermarks could be inserted into rendered images/videos to deter data abuse in practical applications. (3) We caution discretion on behalf of the user, and strongly call for responsible usage of the dataset. For the public good, RenderMe-360 will be open to anyone who
has learned and agrees with our dataset usage agreement, where the data shall be used for research purposes only. Moreover, as our dataset centers on real human data, it is important to consider private information protection and cluster bias problems during the data collection. We summarise the key efforts we made for these aspects as follows - 1) Privacy Authorization and Protection: All subjects have gone through and signed informed consent before data collection. The consent is based on and compliant with the Personal Information Protection Law of the People's Republic of China (PIPL) standards. The users should learn that the dataset is for research purposes only and prohibits harmful or illegal use. Besides, to protect sensitive personal information, we only collect the necessary information of each subject. We anonymize the data by replacing real names with virtual serial numbers. 2) Fairness and Bias: We consider demographic representativeness during data collection and minimize personal bias during annotation. Multiple professional annotators label the same data, and the most accurate result is selected through voting. By implementing these measures and promoting responsible usage, we aim to maximize the positive impact of our dataset while minimizing the potential negative consequences.
Limitations. RenderMe-360 contains over 243 million video frames with high-fidelity captures and corresponding rich annotations. However, there are some potential limitations specifically affecting RenderMe-360: (1) Adequate data volume towards open-world human avatar generation. Despite the significance of our dataset when compared with existing datasets in terms of data volume, diversity, realism, and granularity, etc, there is still a large magnitude gap between in-lab scale collection and internet scale collection. The multi-view capture setting makes it infeasible to reach the same data volume magnitude with internet scale collection like monocular videos or unstructured 2D images. On the one hand, we will scale up the volume in the future dataset version, with continuous effort to collect more human heads. On the other hand, this limitation could be a starting point for future work on developing algorithms via combining both high-quality multiview data and unstructured open-world collection. (2) Constraints in the data capture system. To achieve high-fidelity capture, we apply uniform lighting, 30fps capture speed, and audio collection with the microphone in one position. Upgrading the system could enable relighting, higher speed capture, subtle movement detection, and multi-positional audio collection etc, expanding the possibilities for downstream tasks. (3) There are two potential improvements regarding the data collection setup. Firstly, the naturalness of the wigs in videos may vary due to the random selection process, disregarding the personalized fit for each individual. Insufficiently fitted wigs can lead to a less authentic appearance. To address this, our future endeavors will prioritize the establishment of a more tailored wig collection pipeline, ensuring heightened authenticity. Secondly, an opportunity for improvement lies in expanding our data collection to encompass the depiction of individuals adorned with diverse makeup styles and embellishments. This expansion would augment dataset diversity, bolstering applications in head avatar generation and editing. (4) From a benchmark construction perspective, it is difficult to thoroughly investigate all aspects of the dataset and evaluate all state-of-the-art methods in a single publication. Thus, we regard the benchmark construction upon RenderMe-360 as an ongoing mission. We also welcome contributions from the research community to foster development in this area together.
We build a large-scale 4D human head dataset and relative benchmarks, RenderMe-360, for boosting the research on human head avatar creation. Our dataset covers 500 subjects with diverse appearances, behaviors, and accents. We capture each subject with high-fidelity appearance, dynamic expressions, multiple hairstyles, and various speeches. Furthermore, we provide rich and accurate annotations, which encompass camera parameters, matting, 2D/3D facial landmarks, scans, FLAME fitting, and text descriptions. Upon the dataset, we conduct extensive experiments on the state-of-the-art methods to form a comprehensive benchmark study. The experimental results demonstrate that RenderMe-360 could facilitate downstream tasks, such as novel view synthesis, novel expression synthesis, hair editing, and talking head generation. We hope our dataset can unfold new challenges and provide the cues for future directions of related research fields.
RenderMe-360 is constructed under OpenXDLab – an open platform for X-Dimension high-quality data. This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). It is also partially supported by Singapore MOE AcRF Tier 2 (MOE-T2EP20221-0011, MOE-T2EP20221-0012), NTU NAP.