RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars
Authors: Dongwei Pan
Long Zhuo
Jingtan Piao
Huiwen Luo
Wei Cheng
Yuxin WANG
Siming Fan
Shengqi Liu
Lei Yang
Bo Dai
Ziwei Liu
Chen Change Loy
Chen Qian
Wayne Wu
Dahua Lin
Kwan-Yee Lin
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Head Avatar;Metaverse;Multi-view Human Head Dataset;High-Fidelity;High Diversity;Rich Annotation;Benchmarks
TL;DR: A large-scale digital library for head avatars with three key attributes: high-fidelity, high diversity and rich annotation. 5 benchmarks with 16 methods.

Abstract: Synthesizing high-fidelity head avatars is a central problem for computer vision and graphics. While head avatar synthesis algorithms have advanced rapidly, the best ones still face great obstacles in real-world scenarios. One of the vital causes is the inadequate datasets  -- 1) current public datasets can only support researchers to explore high-fidelity head avatars in one or two task directions; 2) these datasets usually contain digital head assets with limited data volume, and narrow distribution over different attributes, such as expressions, ages, and accessories. In this paper, we present RenderMe-360, a comprehensive 4D human head dataset to drive advance in head avatar algorithms across different scenarios. It contains massive data assets, with 243+ million complete head frames and over 800k video sequences from 500 different identities captured by multi-view cameras at 30 FPS. It is a large-scale digital library for head avatars with three key attributes: 1) High Fidelity: all subjects are captured in 360 degrees via 60 synchronized, high-resolution 2K cameras. 2) High Diversity: The collected subjects vary from different ages, eras, ethnicities, and cultures, providing abundant materials with distinctive styles in appearance and geometry. Moreover, each subject is asked to perform various dynamic motions, such as expressions and head rotations, which further extend the richness of assets. 3) Rich Annotations: the dataset provides annotations with different granularities: cameras' parameters, background matting, scan, 2D/3D facial landmarks, FLAME fitting, and text description.

  
  Based on the dataset, we build a comprehensive benchmark for head avatar research, with 16 state-of-the-art methods performed on five main tasks: novel view synthesis, novel expression synthesis, hair rendering, hair editing, and talking head generation. Our experiments uncover the strengths and flaws of state-of-the-art methods. RenderMe-360 opens the door for future exploration in modern head avatars. All of the data, code, and models will be publicly available at https://renderme-360.github.io/.

RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars
Dongwei Pan
Long Zh Jingtan P Huiwen Luo Wei Cheng!*,
Yuxin Wang!
Siming Fan
Shengqi Liu?
Lei Yang",
Bo Dai',
Ziwei Liu',
Chen Change Loy', Chen Qian',
Wayne Wu
Dahua Lin Kwan- Yeen-Yee Lin
Shanghai AI Laboratory
2 SenseTime Research 3 HKUST
S-Lab, NTU
CUHK 1 linjunyi 93358gma laaaaallcoo coom
(a) High Fidelity
(b) High Diversity
(c) Rich Annotations
Figure I: Overview Me-3er-s--360's core features. We present a large digital asset library RenderMe-360 facilitate development advanced research on high-fidelity head avatar synthesis. It the characteristies (a) high fidelity and (b) high diversity. Also, our dataset comes with (c) rich annotations.
Abstract
Synthesizing high-fidelity head avatars is central problem for computer vision and graphics. While head avatar synthesis algorithms have advanced rapidly, the best ones still face great obstacles in real-world scenarios. One of the vital causes is the inadequate datasets 1) current public datasets researchers to explore high-fidelity head avatars in one or two task directions; 2) these datasets usually contain digital head assets with limited data volume,
only support
'Joint-sccond authors.
37th Conference Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

and narrow distribution over different attributes, such as expressions, ages, and accessories. In this paper, we present RenderMe-360, comprehensive 4D human head dataset to drive advance in head avatar algorithms across different scenarios. It contains massive data assets, with 243+ million complete head frames and over 800K video sequences from 500 different identities captured by multi-view cameras at 30 FPS. It is a large-scale digital library for head avatars with three key attributes: 1) High Fidelity: all subjects are captured in 360 degrees via 60 synchronized, high-resolution 2K cameras. 2) High Diversity: The collected subjects vary from different ages, eras, ethnicities, and cultures, providing abundant materials with distinctive styles in appearance and geometry. Moreover, each subject is asked perform various dynamic motions, such as expressions and head rotations, which further extend the richness of assets. 3) Rich Annotations: the dataset provides annotations with different granularities: cameras' parameters, background matting, scan, 2D/3D facial landmarks, FLAME fitting, and text description.
Based on the dataset, we build a comprehensive benchmark for head avatar re- search, with 16 state-of-the-art methods performed on five main tasks: novel view synthesis, novel expression synthesis, hair rendering, hair editing, and talking head generation. Our experiments uncover the strengths and flaws of state-of- the-art methods. RenderMe- 360 opens the door for future exploration in modern head avatars. All of the data, code, and models will be publicly available at bttpp::/reederree 360. github.io/
1 Introduction
Digitalizing human replicas is a perennial topic in both research and commercial communities, serves as the foundation of many advanced applications, e.g., VR/AR, gaming, and metaverse. Among various tasks, human head avatar synthesis plays crucial but difficult role. This is because the human head performs significant social functions with appearance, expression, speech, etc., which even subtle differences between synthesized and real ones can be casily perceived by human eyes to trigger the uncanny valley effect. How to render, reconstruct, and animate a human head with realism reminds great challenge. Over decades, although numerous approaches have emerged and pushed forward the frontier of facial reconstruction
43 44 and animation [3 33 general full-head level avatar synthesis [56] 3 (33 has only started to actively advance in recent years. Research efforts along human head avatar usually follow the flourishing of deep learning and neural rendering. Such formalizations require large-scale or dense multi-view training datasets to drive progress.
Unlike the efforts on 2D datasets (28 20 which could utilize Internet-scale data enhance the quantity and diversity, the path to constructing a 3D/4D repository is difficult. Thus, current human head-related datasets [5I154] I1] [0] have significant limitations on dataset scale, sample diversity, photorealism, sensory modality, and annotation granularity. For example, Multiface dataset [51] contains facial data with only 13 publicly available subjects, VOCASET [I]] focuses on auditory modality while ignoring other facial functions, and HUMBI Face (54) suffers from low resolution with 2 million pixels. The details of the existing head-related datasets' limitations are shown in Table T These datasets are valuable. However, they can only enable researchers to study small set of problems. The progress of human head avatar algorithms also indicates the saturated performances on existing datasets, while the performance gap between standard datasets and real-world scenarios still remains. Moreover, human head avatar synthesis is complex combination of many fundamental tasks (such as face/head reconstruction, expression animation, and hair modeling/animation)) which requires a comprehensive digital asset library to support the exploration. In nutshell, compared with 2D counterparts, the construction of 3D/4D human head repositories is impoverished.
In this paper, we present Mender: new publicly available large-scale 4D digital asset library with over 243 million frames that features a wide range of downstream tasks, to boost the development of human head avatar creation. RenderMe-360 goes beyond previous datasets in several key aspects: 1) High Fidelity: we set up a high-end data collection system to capture high-resolution raw data of RenderMe-360 With the system, all data is ensured to be captured by 60 cameras at 2448 x 2048 resolution and 30 FPS. 2) High Diversity: We collect 500 different participants, who come from various countries with diverse ages and cultural backgrounds (illustrated in Figure 33). Specifically, about 25% of them are with designed makeup styles and wearing special decorations. such as ancient Chinese makeup styles with delicate hair accessories. These nature differences of the participants provide ample variety in both appearance and accent. For each subject, we capture 12 expressions, 26 or 42 bilingual speeches, and 12 hairstyles (if not specifically required). These collection protocols further enrich the diversity of motion, modality, and appearance. 3) Rich Annotations: We provide
2

---

Table I: Multi-view head datasct comparison. N: Dataset is not released, S: Scanner, M: Mesh, AU: Action
Unit, PF; Per-Frame. Outfit: A variety of clothes-related & accessory-relates designs. Motion: head or body motion, but facial changes not included. Better zoom in for details.
Rese Wig Fader
Appean Amaniacin tedlo Lacket Coppe
STAAA shi moskly
La
Dabset DinRNss 11
Diaasdy
Redion
Grandariiy
(3)
100A312
62118
12M
7
40130
300
C-68
2014
6O DNP
30
7 127
multiple types of annotations with different granularities (Table which ensure the compatibility of one single datasct to various tasks and methods. Concretely, we provide annotations in two levels: per-frame annotations, and per-id annotations. The per-frame annotations refer to annotating every frame of the collected data. These per-frame annotations include camera parameters, matting, facial action units, and 2D/3D landmarks. The per-id annotations refer to annotating keyframes for each identity in fine-grained hierarchy, including 3D scans, FLA fitting, UV maps, and text annotations for appearance and activity descriptions. Our vast exploration space and massive data assets serve as the foundation to investigate the performance boundary of state-of-the-ar head avatar algorithms.
Based on RenderMe-366 dataset, we set up benchmarks on five fundamental tasks, i.e. novel view synthesis, novel expression synthesis, hair editing, hair rendering, and talking head generation, with extensive experimental settings evaluated on 16 baseline methods (Table 2). We probe in detail how different factors might introduce the influences to current baseline methods. Our experiments present many new observations and challenges for the research community to catalyze future research on the human head avatar. We hope MenderMe-360 could kickstart research efforts in related areas, and spur new opportunities not only from our formalized benchmarks, but also alternative ones that the community might come up with from our comprehensive, massive, and publicly available dataset.
2 Related Works
2.1 Multi-View Head Dataset
Data serve the primary fuel for promoting the development of algorithms. While there are many open-world unstructured 2D datasets (20, [59 29 [9] 21 35) or synthetic ones, we focus on those real human heads with structured data. Collecting 3D/4D data is essential for head avatar research in both training and evaluating aspects. In the early days of computer vision, researchers mainly focused on 3D face reconstruction//raccin from data sources that included multi-view cues. In 1999, Blanz and Vetter 3 used a laser scan to capture 3D faces, and proposed to model a morphable model (DMM) from the database. As such a piece of equipment is not suitable for dynamic motion tracking. Zhang et al. (55) present multi-camera active capturing system with six video cameras and two active projectors to ensure spacetime stereo capturing. Later on. Paysan et al. (37 collect 3D faces by ABW-3D system. D3DFACS [10] introduces a dynamic 3D stereo camera system to capture 4D
scans of 10 performers with Action Unit annotations. Upon D3DFACS, Li et al. [22) additionally integrate 4D scans from CAESAR dataset (39] and self-captured ones (from 3dMD system). These datasets are mediocre in texture resolution and quality. Recently, Facescape (52 is proposed to fulfill the raw data quality, in which 3D faces are collected from a dense 68-camera array with 847 subjects performing specific expressions. Whereas, these research efforts are limited to supporting facial shape and expression learning. To take step further on modeling the entire head, i3DMM (53) dataset is proposed with 64 subjects captured by a multi-view scanning system, called Treedys. Since Treedys is not specifically designed for head-scale capture, the authors apply post-process to crop the head meshes based on 3D landmarks, and removing the rest part of the upper body. HUMBI [54] is large-scale dataset, which contains different body part collections. As the systems for these two datasets are not customized to best fit head-level capture, they are limited in resolution. Multiface [51] contains head-oriented collections and detailed annotations, but only releases 13 subjects' data. To facilitate multisensory modeling, VOCASET [II] is proposed. It is a 4D speech-driven scan dataset with about 29 minutes of 4D scans and synchronized audio from 12 speakers. Although VOCASET allows training and testing of spesch-to-animation geometric models and can generalize to new data, it is limited in the extremely narrow diversity of subjects and onefold task. The other alternative is audio-visual data. These datasets are widely used in audio- learning tasks, like lip reading 77 speaker detection [40] and talking head generation [47 (38). For example, GRID [9] and MEAD 47 are sparse multi-view datasets (four and eight respectively). which are characterized by consistent shooting conditions, carefully designed identity, and corpus distribution. However, the sparsity leaves these datasets more often to used in 2D methods.
In contrast, our RenderMe-360, is a large-scale multi-view dataset for high-fidelity head avatar creation research. It is under a head-oriented, and high-resolution data capture environment. It contains diverse data samples (with 500 subjects performing various activities, e.g., expressions,
3

---

PNJOCICY
Data Collection
Annstaaion
posiiom??
cantacss
YES
Canee Collette
PLME
6
8
8
KLAP%
Figure 2: Overview of data collection pipeline. (a) Our system, named as POLICY, records subjects' perfor- mances in 60 different views. (b) Illustration of data collection process. (c) Illustration of annotation pipeline.
speeches, and hair motions), multi-sensory data, and rich annotations. It is comprehensive digital asset library that ensures the compatibility of evaluating multiple head avatar tasks one single dataset. A comparison between RenderMe-360 and other related datasets shown in Table
2.2 Neural Rendering for Head Avatar
Due to space limits, we discuss methods for head rendering in main paper, and unfold the other three directions - hair reconstruction, hair editing, and talking head generation in supplementary.
Head Rendering. Head or face priors often used to condition the neural fields. Such a philosophy
can help either improve the robustness or create controllable avatars [16 2 [5]. Priors like parametric model [2] coefficients, key points, and explicit surface mesh/point clouds are popular ones to be integrated into the framework. For example, NHA [17] presents a framework to learn vertex offsets and attached textures from fitted FLAME surface via coordinate-
MLPs embedded on the surface. NerFACE [15] and IM Avatar (56) use FLAME [22] model expression coefficient to condition the neural field and learn to create an animatable head avatar from monocular video. Taking multi-view images input condition, Neural Volume [30] models dynamic 3D content with volumetric representation. MVP 31 replaces the single volume with mixture of multiple predefined volumetric primitives which improves the resolution and efficiency of volume rendering. To increase the flexibility of re-rendering the avatar in new environments (e.g., novel expression and lighting). PointAvatar [57] presents paradigm of utilizing point-based representation which achieves fast model convergence by coarse-to-
optimization. For generalization, KeypointNeRF [32] synthesizes free viewpoints of human heads via multi-view image features and 3D keypoints. In addition, some researchers use cross-domain data such audio or text to condition the neural fields. For instance, ADNeRF [18] presents 3D-aware alternative to the 2D talking face pipelines by conditioning the radiance field with both head poses and audio fragments.
RenderMe-360
In this section, we introduce RenderMe-366 dataset in detail. We start with the description of our capture system (Section B.Ip. and move an overview process of data collection (Section (.2). Then, we present the data annotation pipeline (Section 3.3). The whole process is visualized in Figure
3.1 Capture System
As illustrated in Figure 2 a). we build a multi-video camera system to record synchronized multi-view videos of human head performance. It contains 60 industrial cameras and covers a field of view of 360 left-to-right and over 160 up-to-down for video capture at the whole-head level. To ensure encompassing fine details (e.g., hair strands, wrinkles, and freckles), we choose cameras with a high resolution at 2448 x 2048. The shutter speed is 30 FPS for capturing fine-grained motion changes. To capture multi-sensory information, condenser microphone is collocated with the camera system, and under the audio-vision synchronization. Please refer to supplementary for more system details.
3.2
Data Collection
The data collection pipeline is illustrated in Figure bb). Specifically, to guarantee the valid rate of captured data, we first apply trial collection with fake head to check on the operability of equipment and adjustment of camera positions before formal acquisition. After this, we start the formal capture process with the following parts for per-person recording: 1) Calibration Capture. We capture camera calibration data before every round of recording. We use a chesshoard and move in front of the cameras fixed-order trajectory. 2) Expression Capture. We ask each subject to
4

---

perform the same expression set, which includes 12 distinctive facial expressions (1 natural and 11 exaggerated expressions) defined in [53]. 3) Hair Capture. To cover diverse hair materials and hair motions, we record 12 video sequences (on average) for each normal subject, with different hairstyles under three levels -original hair, headgear, and wig captures. Specifically, the collected data includes one motion sequence for the subject's 's original hair, one for headgear that hides one's 's hair, and rest sequences for wearing different wigs with random styles and colors. 4) Speech Capture. We provide rich corpus that encompasses single words combined sentences, phonetically balanced protocols, and short paragraphs in two languages (Mandarin and English). For each subject, we randomly pick materials from the corpus and ask the subject to speak 26 or 42 phonetically balanced sentences.
We obtain large-scale dataset of over 800k: recording videos from 500 identities at the end, which is gender-balancee includes multiple ethnicities, and spans ages from to 88 with approximate normal distribution where teenagers and adults form the major part (Figure 3) A more detailed description of our data collection process and related data statistics are discussed in the Supplementary Materials.
3.3 Data Annotation
Diverse and multi-granularity head-related annotations are crucial for the research of human head avatar tasks. However, there is still a deficiency of an all-around head dataset with rich annotation in the research community. To facilitate the development of downstream tasks, we provide rich annotations. We also provide toolbox to automatically label most of the annotations (Figure (c). We unfold the key information in this section. For more details, please refer to Supplementary.
Camera Parameters. We estimate extrinsic matrix and rectify intrinsic matrix for each cam- era via a fine checkerboard pipeline [8]. The process includes checkerboard detection, intrin- sic calibration, and extrinsic calibration with multi-view bundle adjustment. To ensure qual- ity, we additionally applied fast novel view syn- thesis (34], and facial landmark reprojection on multi-view angle-frame to climinate unqualified estimation on camera parameters.
di
2D & 3D Facial Landmarks. 2D landmarks are per-frame detected via an enhanced version of [50] on selected frontal views, which range from 60 left to 60' right. With 2D landmarks from multiple views, RANSAC [14] triangulation is applied to obtain 3D landmarks. To guarantee accuracy, low-quality 2D ones are filtered out with spatial and temporal constraints, and 3D results with large re- re-prjeection errors are also filtered out. For the frames that are neither precisely calculated on 2D landmarks nor 3D landmarks, we manually label the 2D landmarks and re-run the triangulation.
Key data statisties. a) 26 42 speeches recorded per subject. b) Annotation example. c) 12 expressions are captured for each subject. d) 10 wigs (on average) randomly sampled for each normal subject. Original hair and headgear are also captured. Distribution of gender. age, and ethnicity. Better zoom in for details.
Figure 3:
Dense Mesh Reconstruction. Traditional MVS algorithms based on feature points extraction and geometric optimization, such as [46), can only generate irregular point clouds, and have low-quality results in areas of texture missing, such as black hair and dark skin. Therefore, we additionally and optimizes with
apply NeuS [48), which uses neural representation for basede rendering results to do multi-view reconstruction and dense mesh extraction. For video sequences, the first frame is optimized from scratch, then the following frames are fine-tuned on the optimized neural representation to accelerate convergence speed.
Matting. Reasonable foreground segmentation for human heads is challenging. Since diverse hairstyles and accessories form the long-tail problem. Thus, we develop united pipeline that combines video-based matting and scan mesh information to improve matting quality. Specifically, we capture the background prior to each round of recordings, and apply RVM (25) to estimate the rough matting result in the first step. We additionally blend depth-aware mask via Z-buffer during rasterization (26) on the scanned mesh to improve matting quality. With multi-view information, the background ambiguity can be distinguished from other views. We Gaussian Mixture Model to blend the estimations from two models for each pixel 41) For extremely hard cases where both steps cannot output satisfactory results, we add -uman-in-the-lowe labeling.
FLAME Fitting. Since only keyframes are attached with scan meshes to save processing costs, we use two fitting methods in practice- one is fitted with scan mesh and the other is not. For frames with
5

---

scans, we use 3D landmarks to initialize FLAME parameters and optimize via ICP M. Since scan shows accurate facial shape in world space, fitting with it preserves better facial contour. For the ones without scans, personalization is designed. The key is to select neutral frames with corresponding scan geometries, and average across these fitting results to generate a personalized template for each subject. We keep shape parameters constant during fitting the expression sequences which lack scans.
Text Annotation. To facilitate multi-modal research on human head avatars, we provide text descriptions for the captured videos at unprecedented granularity. These descriptions cover both static and dynamic attributes from four major aspects: 1) static facial features of the subjects, where over 90 attributes at general facial appearance, detailed appearance, and lighting condition levels described; 2) static information of non-facial regions, where the texture, material, and shape attributes of subject's 's accessories (such as necklace, earrings, and hairpin) and hairstyle are defined; 3) dynamic facial actions, fine-grain fine-grineed action units (AUs) descriptions based on the FACs system [2] are given; 4) dynamic video activity descriptions, where full-sentence annotations of global action sequence descriptions for each captured video are provided.
4 Benchmark
Our RenderMe-360 dataset provides various potentials in new research directions and applications for human avatar creation. Here, we build a comprehensive benchmark upon RenderMe-360 dataset, with 16 representative methods on five vital tasks of human head avatars (summarized in Table (2) These tasks range from static head reconstruction and dynamic synthesis to generation and editing. For each task, we set up several experimental protocols to probe the performance limits of current otate-of- the-art methods under different settings. Due to space limits, we present the key insights of the novel view synthesis task in the main paper, and highlight the best/worst results in lavender/gray colors. Please refer to Supplementary Materials for benchmarks of the rest four tasks (novel expression synthesis, hair rendering, hair editing, and talking head generation), more implementation details, experiments, qualitativ/quantiattie results, and discussions.
4.1 Novel View Synthesis
Here, we present novel view synthesis (NVS) benchmark of both case-specific (i.e.Single-ID NVS in sub-section 4.1.1) and generalizable (sub-section 4.1.2) tracks.
4.1.1
Single-ID NVS
This case-specific track refers to the setting of training on a single head with multi-view images, which originates from NeRF [3 's de facto setting, to evaluate the robustness of static multi-view head reconstruction. We study four representative methods with two protocols 1 rrotoooolll--// for exploring methods' robustness to different appearance or geometry factors. The dataset is split into three categories, i.e., Normal Case, With Deformable Accessory, and With Complex Accessory, according to the complexity of appearance and geometry. We discuss the protocol in the main paper; roooooooo for probing methods' robustness different camera number and distributions. We discuss this protocol in Supplementary Materials.
Table 2: Methods for RenderMe-360 benchmarks. M: Multi-view images,: St Single-view images, C: Camera Calibration, V: Neural Volumetric, S: Neural SDF, P: Point-based Representation, F: Feature Space, St Static, D: Dynamic, 1: Images Conditioning, L: Latent Codes, P. Parametric Models K: Face Keypoints. Radiance Field-based, F: SaF-based, N: Convolution-basel S: Case specific, G: Generalizable, Better zoom in.
Instant-NC NGP
MAV
18
IR Keypann KeypaniNeBBa
VisioN NoRFace
Point AvattAv
NeRF
ADNORF
NeuS
MM
NSPF
NR.
Har
Atiribule
Novel View Symthesis
Novel Expression Synthesis
Hair Rendering
Hair Editing
Talking Head
Required Dara
M+C
M+C
MrC
MFC
M-C
M+C
StC
S+C
S=C
M+C
M-C
S
S+C
Representation
Stanic/Dnamma
S
D
D
S
D
D
D
S
Conditioning
I
1
I
I
I
L
L
L
L
Face Priors
X
P
K
X
P
P
X
F
3D Consistency
R
Generalizability
S
S
G
G
S
S
G
G
Settings. We select 20 identities from the three categories to evaluate the methods. For the training- testing split, we uniformly sample 22 views from all 60 views as test views, and use the rest camera views to train each model. A visualization of camera distribution is shown in Supplementary Materials,

---

noted as Cam1. The four methods for comparison are: Instant-NGP 34, NeuS 48 NV 130 and MVP [II The first three methods are originally designed for ceneera--urpose rase-specific NVS. and the last one is designed for human head avatar reconstruction. We compute PSNR, SSIM, and LPIPS for rendered novel view images against ground-truth. The quantitative results are listed in Table 3 and the qualitative ones are listed in Supplementary Materials.
Results. We observed three key phenomena under #Protocol-1: 1) All methods tend to drop the performance lengthways along the Table 3 with the level of accessory complexity increasing. This phenomenon reflects the status quo that we do not yet have one strong paradigm for robust case- specific human head multi-view reconstruction. 2) NeuS yields the best performance on average. There are two possible underlying reasons. First, NV and MVP are dynamic methods while not emphasizing cemporal-consistenge constraints. Thus, when comparing these methods with static ones under static measurement, the perturbation of data sequences would affect these two methods' construction dynamic fields to certain degrees. Second, by associating the quantitative results with the qualitative ones, we can find that NeuS performs well in global shape reconstruction with almost no surrounding noise due to its surface representation property, but has a much smoothing surface appearance. In contrast, Instant-NGP and MVP can recover better high-freee high-frequency details. MVP uses multiple- primmiive representation with different networks to render, equipping the model with larger representative capacity. Whereas, they produce more surrounding noise. Neural Volume renders images mostly with artifacts. We could draw the idea that surface representation helps the novel view reconstruction in a global fhrme-forming manner. 3) NV suffers from limited grid resolution (although it uses inverse warping to ease the problem) and inaccurate alpha value estimation. Thus, it introduces more artifacts than others, and strenuous in reconstructing high-frequency details.
Table 3: Single ID NVS (fProtocol-I). We evaluate methods under three subsets with levels of complexity.
Split
Metries
NGP
[10]
MVP
Split
Metries
NGP
NeuS [8]
[0]
MVV [1]
Normal Case
LPIPS.
PSNRI
SSIM
24.71
0.845
0.28
26.29
0.927
0.31
0.777
19:61
0.29
0.895
23.65
0.14
With Complex
Accrssory
LPIPS]
PSNRT
SSIM
0.776
20.54
0.36
22.89
0.874
0.16
16:46
0,508
0.44
0.83
0.18
21.5
With Deformaalle Accessory
PSNR1
SSIM
23,06
0,807
23.53
0:904
0.703
17.85
0.993
23,93
Overall
PSNR1
SSIM
23.21
0.819
0.906
24.67
18:56
0.723
0.876
23.1
LPIPS.
0.31
0.13
0.36
0.12
LPIPS
0.31
0.13
0.33
0.15
4.1.2 Generalizable NVS
This track refers to the setting of training across multiple human heads, and testing on unseer" T human heads (i.e., new identitics) or unseen motions (e.g., expressions) with conditioning on one or few input images (as source views). It allows us to evaluate the network's 's effectiveness in learning priors, and the ability to adapt priors. We investigate three methods under two protocols in the main 1) #Protocol- for investigating methods' generalization ability on geometry deformation via evaluating the generalization ability to unseen expressions on seen identities. 2) #Protocol-2 for probing methods' capability in learning category-level human head priors via evaluating the generalization ability to unseen identities. We also name this protocol as Unseen ID) NVS; This setting is challenging as requires the model to generalize to both new appearances and geometries. To further reveal the factors that might have influences on generalization, we enrich both protocols with four sets of training-teste testiig view settings and three data subsets under different complexity.
Settings. We study three generalizable methods: IBRNet [49]. VisionNeRF (24), and Keypoint- NeRF (32. For teiningg-esstig identity split, we select a subset from RenderMe-366 with 160 identities for training and 20 for serving as unseen identities. The selected identities are evenly sampled from the three data subsets. We select 7 out of 60 camera views novel views. Note that, we calculate the metries in #Protocol-2 on all expressions. As a consequence, 10 expression structures attached with trained identities are covered in training set, and the rest 2 expression structures are unseen. Such an evaluation strategy provides the feasibility for researchers to analyze their methods' generalization ability on appearance and geometry in both entangled and disentangled aspects.
Results. The quantitative results are shown in Table 4 From per method perspective, draw the consistent conclusions that: 1) random view training could help enhance the model's robustness on both unseen expression and identity tasks; 2) the performance declines in terms of most metrics with the complexity of human head's 's appearance/geometry increase; 3) the Unseen ID NVS task introduces larger performance drop rate than Unseen Expression NVS. These two phenomena suggest that these generalizable methods could learn priors like the information of 'minimal-accessee
Note that the adjective 'seen" refers to the sample that is used in training, and 'unscen' means the sample not used as training data.
7

---

Table 4: Benchmark results on generalizable NVS. The results are evaluated on: (1) unseen expressions of
sampled training identities, and (2) unseen identities in each test split. (LPIPS* denotes LPIPS x 1000)
Trvining Setting
Testing Setting
Matthele
Nermal Cae
With Infiemsuble
Accossetis
With Complay Accemario
Osenail
Unsen Expessio NYS
23.15
146.0*
20A32
13.40
199.8
20.33
0.322
157.57
22.57
0.515
165.41
Fixed Niews
VilooNeRF 123
23.57
0.986
135.52
20.42
0.546
186.04
30.83
0.435
182 00
22.11
0.813
063
Pood Nlrae
24:71
155:4
21231
137151
0/212
172:5
07313
112.37
Kasdorn View
15.99
01914
148:20
21.43
0.983
20.37
08:07
152.00
0.893
43
256.5
1IESS 11g
25377
03918
008
21134
112137
19715
19.20
03301
192.05
03.33
03377
18:15
Kasdor: Views
Raslies Vices
KoyourNage
VaRsNNe
18:74
28:18
00924
0.900
179.71
113.5
21:00
16:05
0.517
11970
153.66
17.02
23.00
0.775
02011
170
182.08
18:04
0290
6.850
0381
19119
148.0 148.19
KesporeNert
18.55
0.807
134.33
17.00
1.615
190.04
10.65
0.776
195 67
17.45
0.947
196.44
Trining Scttan
Ballig Settiax
Michods
PSNNK
Nonual Cape
381M1
1190%
W500 Deborasshic Accessero Rm
P5NKT
1
15N4KT
Coungees Accesonres
5SU341
15/4K1
551M1
Orondl
IBRNk 10
255.06
XVS
Food Views
VasseNand
21.00
11.06
146.44
18:00
03911
218-23
17:36
0.255
282 mm
19:34
0.817
192.55
Fizd Niowa
22.54
11397
155.00
19:74
0X31
103:75
12:12
0.251
207.23
07531
19970:
Rustor Vieas
VesezNund
24.01
[1815
15.13
0,57
198.45
19:35
0.29
197.42
0.813
171.9
Uitter 110
22:24
11305
18:27
0331
21733
12:0
0.236
256.0
33-13
0:510
198-77
Fioel Views
Bandon Views
15R25 (00
2233
18.06
12977
116.31
15605
16.15
18.05
0.990
0338
18541
195114
16.19
18:10
0.208
0.220
250.09
230-22
17.55
20/23
0.855
171.94
155.27
KeyoaasNeti
18.02
0.66
145.30
15.75
0.794
194.16
16.15
0.742
227.40
16.99
0.11
178.06
IBRNet
KeyPointNeRF VisionNeRF Ground Truth
IBRNet
KeyPoinlNeRF VisionNeRF Ground Truth
Unseon Expression
Unseon
Figure 4: Qualitative Results of Generalizable NVS (#Protocol-1&2). We illustrate three generalizable methods in two different settings. The regions in red boxes are zoomed in for better visualization.
mean head, and local geometry transformation on a certain level, while still struggling with more diverse scenarios that are long-tail distributed (please associate statistical details in Supplementary Materials). In addition, there are several interesting observations when comparing the three methods: 1) VisionNeR achieves the best results on average. The robustness might come from large capacity of learnable variables from a transformer-based structure on image features and the multi- resolution based encoder. 2) IBRNet 49 results in blurry synthesis even under the train and test settings on fixed views. 3) KeypointNeRF 332 falls behind for most of the scenarios, but in the lead on LPIPS on average. In other words, KeypointNeRF benefits in perceptual measurement like LPIPS while suffering from pixel-wise measurements. We infer the possible reason behind the contradictory metric performances is that - the modules driven by triangulated keypoints provide better feature and view alignments in an explicit manner to help reconstruct the radiance fields. Whereas, such a key insight is a double-edged sword for full human head tasks. Since only the facial region could be well guaranteed with accessible facial landmarks. As a consequence, non-facial regions, like the hat in Figure 4 more blurry than the facial region and distorted in the geometry aspect. Moreover, KeypointNeRF only renders the intersected frustum regions from source views in practice, which aggravates the performance problem from the full-head mcasurements. The results turn better when we only calculate regions that KeypointNeRF could render, as shown in Supplementary Materials.
4.2 Intra-Datass Evaluation
Settings. To demonstrate the strengths of our dataset's diversity and scale, we first conduct an intra-dataset evaluation with different subset settings. Concretely, we separate the training data into three subsets according to accessories difficulliess, namely. 'Subset 1-3' which corresponds to "Normal Case', 'With Deformable Accessories", and 'With Complex Accessories" respectively. We also randomly sample subsets with 30% and 50% of the training data to investigate the effectivvnness of the data scale. We train models three generalizable methods [24] B2) on different subsets, and follow the strategy of #Protocol-2 in Sec. 441.2 To ensure faimesss we stop model training on the same global step with the same learning rate.
Results. We span the results of models trained on different training sets by test splits. The quan- titative results of IBRNet [49arr visualized in Fig. 5 Please refer to supplementary for detailed numbers and qualitative results. Generally, models trained on one subset achieve relatively better performance on test split of the same category than the ones train other subsets. For instance, the
8

---

model trained on 'Subset I' gets the best performance in "Normal Case" splits while performing unsatisfactorily in other splits with accessories. With the full training data set, the model tends to overfit less on specific splits and achieves better overall performance. For models trained on smaller random datasets with even distributed data coverage, there is no obvious overfitting on any certain split, While we can observe that with larger scale of data, the model generalizes better on novel identities. To sum up, believe the proposed dataset could improve the robust- ness of generalization methods with both the large scale and the wide range of data distribution.
Stated APSNN
Soaked 1S500
4.3 Cross-Dataset Evaluation
Settings. We conducted cross-dataset experiments to vali- date the transferability of our dataset. Specifically, we use Multiface dataset (51) for comparison. Similar to intra- dataset evaluation, three aforementioned generalization meth- ods 449 24 32) used in Section are utilized to validate the cross domsin performance of our proposed RenderMe- and Multiface. We rigidly transform the Multiface's camera system to align with our dataset so as to avoid coordinate off- sets. For fairness, we also crop and resize both source and target images with 512 512, and train models with the same global step and learning rate. We also follow the strategy of #Protocol-2 in Sec, 4.1.2 models trained on both datasets are directly evaluated on test sets without any fine-tuning.
Selwor
Satwe?
Bonks
Oweii
Roadie 206
Casples
7dll St:
Accssoiis
With Detommabe
Ascesodis
Figure 5: Intra- evaluation. We visualize quantitative results by training and testing on different subsets of the proposed dataset. Smaller bubbles indi- cate better performances. Better zoom in for details.
Results. We visualize the cross-dataset performance of three methods Fig. 6 where we can observe that our dataset achieves excellent cross-domain performance in all three gen- eralizable methods [32 49 24 Compared to RenderMe-360) Multiface dataset only contains 13 identities. Thanks to the large data coverage and scale of RenderMe-360, it enables robust generalization on novel identi- ties in both in-domain and cross-domain settings. Especially, when testing on the Multiface test set, the cross-domain models trained our dataset even outperform the in-domain evaluation of Multiface trained models.
5 Boarder Impact and Limitations
Previously, the unavailability of publicly accessible datasets comprising large-scale high-fidelity human heads had im- peded advancements in related fields for years. The pro- posed RenderMe-360 dataset, together with the comprehensive benchmark, is expected to advance the research community in developing high-fidelity head avatars. It also brings new opportunities to many downstream applications in various fields, such as video telephony and the film industry, which shall be beneficial for society. However, there are still several challenges beyond the current scope of this project, that are essential tackle in the future. We discuss the challenges from both ethical consideration and dataset limitation aspects in the following subsections.
Figure 6: Cross-Dataset evaluation. We visualize the quantitative results by training and testing on the different datasets. Better zoom in for details.
Ethical Consideration. Both our dataset and future research efforts based on it, might be applied for malicious purposes like identity theft and fake news. For example, one might use the data to generate fake videos performing various talking interactions with realistic appearances, head/facial/lip motions, and audio. Then, the generated videos might be used in illegal activities such as financial fraud and emotional harm. This might lead to social instability and inflict emotional harm. To alleviate the negative impacts, there are three doable solutions.
(1) DeepFake detection algorithms could be applied to evaluate the realism of videos. In this way, the public could be informed whether the video content is real or fake. Besides, our dataset, as well as the data synthesized by algorithms (from our benchmarks), could also serve references for advancing the development of deepfake detection algorithms.
(2) Watermarks could be inserted into rendered images/videon to deter data abuse in practical applications. (3) We caution discretion on behalf of the user, and strongly call for responsible usage of the datasct. For the public good, RenderMe-366 will be open to anyone who

---

has learned and agrees with our dataset usage agreement. where the data shall be used for research purposes only. Moreover, our dataset centers real human data, it is important to consider private information protection and cluster bias problems during the data collection. We summarise the key efforts we made for these aspects as follows- - 1) Privacy Authorization and Protection: All subjects have gone through and signed informed consent before data collection. The consent is based on and compliant with the Personal Information Protection Law of the People's Republic of China (PIPL) standards. The users should learn that the dataset is for research purposes only and prohibits harmful or illegal use. Besides, to protect sensitive personal information, we only collect the necessary information of each subject. We anonymize the data by replacing real names with virtual serial numbers. 2) Fairness and Bias: We consider demographic representativeness during data collection and minimize personal bias during annotation. Multiple professional annotators label the same data, and the most accurate result is selected through voting. By implementing these measures and promoting responsible usage, aim to maximize the positive impact of our dataset while minimizing the potential negative consequences.
Limitations. RenderMe-360 contains over 243 million video frames with high-fidelity captures and corresponding rich annotations. However, there are some potential limitations specifically affecting RenderMe-360: (1) Adequate data volume towards open-world human avatar generation. Despite the significance of our dataset when compared with existing datasets in terms of data volume, diversity, realism, and granularity, etc, there is still large magnitude gap between in-lab scale collection and internet scale collection. The multi-view capture setting makes it infeasible to reach the same data volume magnitude with internet scale collection like monocular videos or unstructured 2D images. On the one hand, we will scale up the volume in the future dataset version, with continuous effort to collect more human heads. On the other hand, this limitation could be a starting point for future work developing algorithms via combining both high-quality multiview data and unstructured open-word collection. (2) Constraints the data capture system. To achieve high-fidelity capture, we apply uniform lighting. 30fps capture speed, and audio collection with the microphone in one position. Upgrading the system could enable relighting. higher speed capture, subtle movement detection, and multi-positional audio collection etc, expanding the possibilities for downstream tasks. (3) There are two potential improvements regarding the data collection setup. Firstly, the naturalness of the wigs in videos may vary due to the random selection process, disregarding the personalized fit for each individual. Insufficiently fitted wigs can lead to less authentic appearance. To address this, our future endeavors will prioritize the establishment of more tailored wig collection pipeline, ensuring heightened authenticity. Secondly, an opportunity for improvement lies in expanding our data collection to encompass the depiction of individuals adorned with diverse makeup styles and embellishments. This expansion would augment dataset diversity, bolstering applications in head (4) From a benchmark construction perspective, it is difficult to thoroughly investigate all aspects of the dataset and evaluate all state-of-the-art methods in a single publication. Thus, we regard the benchmark construction upon RenderMe-360 an ongoing mission. We also welcome contributions from the research community to foster development in this area together.
avatar generation and editing.
6 Conclusion
We build a large-scale 4D human head dataset and relative benchmarks, RenderMe-360 for boosting the research on human head avatar creation. Our dataset covers 500 subjects with diverse appearances, behaviors, and accents. We capture each subject with high- sighliideiity appearance, dynamic expressions, multiple hairstyles, and various speeches. Furthermore, we provide rich and accurate annotations. which encompass camera parameters, matting, 2D/3D facial landmarks, scans, FLAME fitting, and text descriptions. Upon the dataset, we conduct extensive experiments on the state-of-the of-thh-art methods to form comprehensive benchmark study. The experimental results demonstrate that RenderMe-360 could facilitate downstream tasks, such as novel view synthesis, novel expression synthesis, hair editing, and talking head generation. We hope our dataset can unfold new challenges and provide the cues for future directions of related research fields.
A Acknowledgment
RenderMe-360 is constructed under OpenXDLab- an open platform for X-Dimension data. This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). It is also partially supported by Singapore MOE AcRF Tier 2 (MOE-T2EP20221-001,, -0012), NTU NAP.
10

---

References
[1] K Somani Arun, Thomas Huang, and Steven D Blostein. Least-squares fitting of two 3-d point sets. TPAMI, 1987.
[2] ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu, Rignerf: Fully controllable neural 3d portraits. In CVPR, 2022.
[3] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In CGIT, 1999.
[4] Volker Blanz and Thomas Vetter. Face recognition based fitting 3d morphable model. TPAMI, 2003.
[5] Chen Cao, Tomas Simon, Jin Kyu Kim, Gabe Schwartz, Michael Zollhoefer, Shun-Suke Saito, Stephen Lombardi, Shih-En Wei, Danielle Belko, Shoou-I Yu, et al. Authentic volumetric avatars from a phone scan. TOG.
[6] Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In CVPR, 2017.
[7] Joon Son Chung and Andrew Zisserman. Lip reading in the wild. In ACCV, 2016.
[8] XRPrimer Contributors. Openxrlab foundational library for xr-related algorithms. 2023.
https
[9] Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao. An audio-visual corpus for speech perception and automatic speech recognition. JASA, 2006.
[10] Darren Cosker, Eva Krumhuber, and Adrian Hilton. A facs valid 3d dynamic action unit database with applications 3d dynamic morphable facial modeling. In ICCV, 2011.
[11] Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J Black. Capture, learning, and synthesis of 3d speaking styles. In CVPR, 2019.
[12] Paul Ekman and Wallace V Friesen. Facial action coding system. EPNB, 1978.
[13] Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart. Learning an animatable detailed 3d face model from in-the-wild images. TOG, 2021.
[14] Martin A. Fischler and Robert C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 1981.
[15] Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In CVPR, 2021.
[16] Xuan Gao, Chenglai Zhong, Jun Xiang, Yang Hong. Yudong Guo, and Juyong Zhang. Recon- structing personalized semantic facial nerf models from monocular video. TOG, 2022.
[17] Chilip-Willian Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. Neural head avatars from monocular rgb videos. In CVPR, 2022.
[18] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In ICCV, 2021.
[19] Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned- Miller. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on FRLI, 2008.
[20] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[21] Ira Saaetmaachhh-SSSzemmaae Steven M Seitz, Daniel Miller, and Evan Brossard. The megaface benchmark: million faces for recognition at scale. In CVPR, 2016.
[22] Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4D scans. TOG, 2017.
[23] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.

---

[24] Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung- Yi Lin, Yi-Chang Shih, and Ravi Ramamoor- thi. Vision transformer for nerf-based view synthesis from a single input image. In WACV, 2023.
[25] Shanchuan Lin, Linjie Yang, Imran Saleemi, and Soumyadip Sengupta. Robust high-resolution video matting with temporal guidance. In WACV, 2022.
[26] Shichen Liu, Tianye Li. Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In ICCV, 2019.
[27] Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou. Semantic-aware implicit neural audio-driven video portrait generation. In ECCV, 2022.
[28] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
[29] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. RA, 2018.
[30] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv, 2019.
[31] Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering. TOG, 2021.
[32] Marko Mihajlovic, Aayush Bansal, Michael Zollhoefer, Siyu Tang, and Shunsuke Saito. Key- pointnerf:: Generalizing image-based volumetric avatars using relative spatial encoding of keypoints. In ECCV, 2022.
[33] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[34] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. TOG, 2022.
[35] Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. 2015.
[36] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV, 2021.
[37] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. In ICACSBS, 2009.
[38] Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando De la Torre, and Yaser Sheikh. Meshtalk: 3d face animation from speech using cross-modality disentanglement. In ICCV, 2021.
[39] Kathleen Robinette, Sherri Blackwell, Hein Daanen, Mark Bochmer, and Scott Fleming. Civil- ian american and european surface anthropometry resource (caesar), final report. volume 1. summary. 2002.
[40] Joseph Roth, Sourish Chaudhuri, Ondrej Klejch, Radhika Marvin, Andrew Gallagher, Liat Kaver, Sharadh Ramaswamy, Arkadiusz Stopezynski, Cordelia Schmid, Zhonghua Xi, et al. Ava active speaker: An audio-visual dataset for active speaker detection. In ICASSP, 2020.
[41] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "ggabbuu" interactive foreground extraction using iterated graph cuts. TOG, 2004.
[42] Wei Tianyi, Chen Dongdong, Zhou Wenbo, Liao Jing, Tan Zhentao, Yuan Lu, Zhang Weiming, and Yu Nenghai. Hairclip: Design your hair by text and reference image. In CVPR, 2022.
[43] Luan Tran and Xiaoming Liu. Nonlinear 3d face morphable model. In CVPR, 2018.
[44] Luan Tran and Xiaoming Liu. On learning 3d face morphable model from in-the-wild images. TPAMI, 2019.
12

---

[45] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In ICCV, 2021.
[46] Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Pablo Speciale, and Marc Pollefeys. Patchmatchnet: Learned multi-view patchmatch stereo. 2020.
[47] Kaisiyuan Wang. Qianyi Wu, Linsen Song, Zhuogian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking-face generation. In ECCV, 2020.
[48] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv, 2021.
[49] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In CVPR, 2021.
[50] Wayne Wu, Chen Qian, Shuo Yang. Quan Wang. Yici Cai, and Qiang Zhou. Look at boundary: A boundary-aware face alignment algorithm. In CVPR, 2018.
[51] Cheng-hsin Wuu, Ningyuan Zheng, Scott Ardisson, Rohan Bali, Danielle Belko, Eric Brock- meyer, Lucas Evans, Timothy Godisart, Hyowon Ha, Alexander Hypes, et al. Multiface: A dataset for neural face rendering. arXiv, 2022.
[52] Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao. Facescape: scage-scale high quality 3d face dataset and detailed riggable 3d face prediction. In CVPR, 2020.
[53] Tarun Yenamandra, Ayush Tewari, Florian Bernard, Hans-Peter Seidel, Mohamed Elgharib, Daniel Cremers, and Christian Theobalt. i3dmm: Deep implicit 3d morphable model of human heads. In CVPR, 2021.
[54] Zhixuan Yu, Jae Shin Yoon, In Kyu Lee, Prashanth Venkatesh, Jaesik Park, Jihun Yu, and Hyun Soo Park. Humbi: A large multiview dataset of human body expressions. In CVPR, 2020.
[55] Li Zhang. Noah Snavely, Brian Curless, and Steven M. Scitz. Spacetime faces: high resolution capture for modeling and animation. TOG, 2004.
[56] Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C Bühler, Xu Chen, Michael Black, and Otmar Hilliges. Im avatar: Implicit morphable head avatars from videos. CVPR, 2022.
[57] Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J Black, and Otmar Hilliges. Pointa- vatar: Deformable point-based head avatars from videos. arXiv, 2022.
[58] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adversarially disentangled audio-visual representation. In AAAI, 2019.
[59] Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv-hq: A large-scale video facial attributes dataset. In ECCV, 2022.
13


Reviews:
[{"review_id": "AWiwv7kP6y", "reviewer": "Reviewer_CynN", "summary": "", "questions": "", "limitations": "Weaknesses:\n1. The dataset covers 500 different identities. I wonder how many old people and children are included? Additionally, I am curious about how the race is distributed across the dataset.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "\nStrengths:\nThe paper is well-written and easy to understand.\nThe built dataset is very large with high fidelity, high diversity, and rich annotations.\nThis paper evaluates 16 state-of-the-art methods performed on five main tasks: novel view synthesis, novel expression synthesis, hair rendering, hair editing, and talking head generation.\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "lzzZ8ty4I8", "reviewer": "Reviewer_8MTN", "summary": "", "questions": "", "limitations": "potential negative societal impact is not discussed ; the limitations of the dataset is not discussed either. ", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The proposed dataset features a number of strengths in comparison with the state-of-the-art datasets:\n1) large diversity: 500 subjects, different ages, ethnicities, cultures\n2) full coverage and high quality: subjects are captured under 360 degrees using 60 synchronized 2K cameras\n3) rich annotations, including cameras' parameters, 2D/3D facial landmarks, and text description 90 attributes \n4) design of a comprehensive benchmark based on RenderMe-360 dataset, including novel view synthesis, novel expression synthesis, etc, and evaluation of several state-of-the-art algorithms for each benchmark task", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "7ArbBFAgvT", "reviewer": "Reviewer_q6aZ", "summary": "", "questions": "", "limitations": "I suggest the authors add more discussion on the ethical consideration of the dataset in the main paper.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The authors provide detailed data capture and processing pipeline for reference and the dataset holds enough diversity. Moreover, they present a comprehensive benchmark using the collected data with 16 methods, showing the potential of the dataset on various tasks of human head avatars. As the authors promise to make all the data, code, models, and annotations public, I believe that the dataset can facilitate the development of the community.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper creates a diverse, high-quality dataset for generating human head avatars called RenderMe-360. The details are very rich that can benefit the entire imaging community.  The built dataset is very large with high fidelity, high diversity, and rich annotations. Overall, all reviewers give positive comments, thus the area chair suggests to accept it.
Decision: Accept (Poster)