SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dongwei Pan<sup>1</sup>, Long Zhuo<sup>1*</sup>, Jingtan Piao<sup>2,5*</sup>, Huiwen Luo<sup>1*</sup>, Wei Cheng<sup>1*</sup>, Yuxin Wang<sup>1,3*</sup>,
Siming Fan $^{2}$ , Shengqi Liu $^{2}$ , Lei Yang $^{2}$ , Bo Dai $^{1}$ , Ziwei Liu $^{4}$ , Chen Change Loy $^{4}$ , Chen Qian $^{1}$ ,
Wayne Wu $^{1}$ , Dahua Lin $^{1,5}$ , Kwan-Yee Lin $^{1,5}$
$^{1}$  Shanghai AI Laboratory
$^{2}$  SenseTime Research
$^{3}$  HKUST
4 S-Lab, NTU
5 CUHK
linjunyi9335@gmail.com
(a) High Fidelity
Figure 1: Overview of RenderMe-360's core features. We present a large digital asset library RenderMe-360 to facilitate the development of advanced research on high-fidelity head avatar synthesis. It has the characteristics of (a) high fidelity and (b) high diversity. Also, our dataset comes with (c) rich annotations.
(c) Rich Annotations
(b) High Diversity
Synthesizing high-fidelity head avatars is a central problem for computer vision and graphics. While head avatar synthesis algorithms have advanced rapidly, the best ones still face great obstacles in real-world scenarios. One of the vital causes is the inadequate datasets - 1) current public datasets can only support researchers to explore high-fidelity head avatars in one or two task directions; 2) these datasets usually contain digital head assets with limited data volume,
*Joint-second authors.
and narrow distribution over different attributes, such as expressions, ages, and accessories. In this paper, we present RenderMe-360, a comprehensive 4D human head dataset to drive advance in head avatar algorithms across different scenarios. It contains massive data assets, with  $243+$  million complete head frames and over  $800k$  video sequences from 500 different identities captured by multi-view cameras at 30 FPS. It is a large-scale digital library for head avatars with three key attributes: 1) High Fidelity: all subjects are captured in 360 degrees via 60 synchronized, high-resolution  $2K$  cameras. 2) High Diversity: The collected subjects vary from different ages, eras, ethnicities, and cultures, providing abundant materials with distinctive styles in appearance and geometry. Moreover, each subject is asked to perform various dynamic motions, such as expressions and head rotations, which further extend the richness of assets. 3) Rich Annotations: the dataset provides annotations with different granularities: cameras' parameters, background matting, scan, 2D/3D facial landmarks, FLAME fitting, and text description.
Based on the dataset, we build a comprehensive benchmark for head avatar research, with 16 state-of-the-art methods performed on five main tasks: novel view synthesis, novel expression synthesis, hair rendering, hair editing, and talking head generation. Our experiments uncover the strengths and flaws of state-of-the-art methods. RenderMe-360 opens the door for future exploration in modern head avatars. All of the data, code, and models will be publicly available at https://renderme-360.github.io/.
Digitalizing human replicas is a perennial topic in both research and commercial communities. It serves as the foundation of many advanced applications, e.g., VR/AR, gaming, and metaverse. Among various tasks, human head avatar synthesis plays a crucial but difficult role. This is because the human head performs significant social functions with appearance, expression, speech, etc., in which even subtle differences between synthesized and real ones can be easily perceived by human eyes to trigger the uncanny valley effect. How to render, reconstruct, and animate a human head with realism reminds a great challenge. Over decades, although numerous approaches have emerged and pushed forward the frontier of facial reconstruction [43, 44] and animation [13, 38], general full-head level avatar synthesis [56, 3, 53] has only started to actively advance in recent years. Research efforts along human head avatar usually follow the flourishing of deep learning and neural rendering. Such formalizations require large-scale or dense multi-view training datasets to drive progress.
Unlike the efforts on 2D datasets [28, 20], which could utilize Internet-scale data to enhance the quantity and diversity, the path to constructing a 3D/4D repository is difficult. Thus, current human head-related datasets [51-54, 11, 10] have significant limitations on dataset scale, sample diversity, photorealism, sensory modality, and annotation granularity. For example, Multiface dataset [51] contains facial data with only 13 publicly available subjects, VOCASET [11] focuses on auditory modality while ignoring other facial functions, and HUMBI Face [54] suffers from low resolution with 2 million pixels. The details of the existing head-related datasets' limitations are shown in Table 1. These datasets are valuable. However, they can only enable researchers to study a small set of problems. The progress of human head avatar algorithms also indicates the saturated performances on existing datasets, while the performance gap between standard datasets and real-world scenarios still remains. Moreover, human head avatar synthesis is a complex combination of many fundamental tasks (such as face/head reconstruction, expression animation, and hair modeling/animation), which requires a comprehensive digital asset library to support the exploration. In a nutshell, compared with 2D counterparts, the construction of 3D/4D human head repositories is impoverished.
In this paper, we present RenderMe-360, a new publicly available large-scale 4D digital asset library with over 243 million frames that features a wide range of downstream tasks, to boost the development of human head avatar creation. RenderMe-360 goes beyond previous datasets in several key aspects: 1) High Fidelity: we set up a high-end data collection system to capture high-resolution raw data of RenderMe-360. With the system, all data is ensured to be captured by 60 cameras at  $2448 \times 2048$  resolution and 30 FPS. 2) High Diversity: We collect 500 different participants, who come from various countries with diverse ages and cultural backgrounds (illustrated in Figure 3). Specifically, about  $25\%$  of them are with designed makeup styles and wearing special decorations, such as ancient Chinese makeup styles with delicate hair accessories. These nature differences of the participants provide ample variety in both appearance and accent. For each subject, we capture 12 expressions, 26 or 42 bilingual speeches, and 12 hairstyles (if not specifically required). These collection protocols further enrich the diversity of motion, modality, and appearance. 3) Rich Annotations: We provide
multiple types of annotations with different granularities (Table 1), which ensure the compatibility of one single dataset to various tasks and methods. Concretely, we provide annotations in two levels: per-frame annotations, and per-id annotations. The per-frame annotations refer to annotating every frame of the collected data. These per-frame annotations include camera parameters, matting, facial action units, and 2D/3D landmarks. The per-id annotations refer to annotating keyframes for each identity in fine-grained hierarchy, including 3D scans, FLAME fitting, UV maps, and text annotations for appearance and activity descriptions. Our vast exploration space and massive data assets serve as the foundation to investigate the performance boundary of state-of-the-art head avatar algorithms.
Based on RenderMe-360 dataset, we set up benchmarks on five fundamental tasks, i.e., novel view synthesis, novel expression synthesis, hair editing, hair rendering, and talking head generation, with extensive experimental settings evaluated on 16 baseline methods (Table 2). We probe in detail how different factors might introduce the influences to current baseline methods. Our experiments present many new observations and challenges for the research community to catalyze future research on the human head avatar. We hope RenderMe-360 could kickstart research efforts in related areas, and spur new opportunities not only from our formalized benchmarks, but also alternative ones that the community might come up with from our comprehensive, massive, and publicly available dataset.
2.1 Multi-View Head Dataset
Data serve as the primary fuel for promoting the development of algorithms. While there are many open-world unstructured 2D datasets [20, 59, 29, 19, 21, 35] or synthetic ones, we focus on those real human heads with structured data. Collecting 3D/4D data is essential for head avatar research in both training and evaluating aspects. In the early days of computer vision, researchers mainly focused on 3D face reconstruction/tracking from data sources that included multi-view cues. In 1999, Blanz and Vetter [3] used a laser scan to capture 3D faces, and proposed to model a morphable model (i.e., 3DMM) from the database. As such a piece of equipment is not suitable for dynamic motion tracking, Zhang et al. [55] present a multi-camera active capturing system with six video cameras and two active projectors to ensure spacetime stereo capturing. Later on, Paysan et al. [37] collect 3D faces by ABW-3D system. D3DFACS [10] introduces a dynamic 3D stereo camera system to capture 4D high-quality scans of 10 performers with Action Unit annotations. Upon D3DFACS, Li et al. [22] additionally integrate 4D scans from CAESAR dataset [39] and self-captured ones (from 3dMD system). These datasets are mediocre in texture resolution and quality. Recently, Facescape [52] is proposed to fulfill the raw data quality, in which 3D faces are collected from a dense 68-camera array with 847 subjects performing specific expressions. Whereas, these research efforts are limited to supporting facial shape and expression learning. To take a step further on modeling the entire head, i3DMM [53] dataset is proposed with 64 subjects captured by a multi-view scanning system, called Treedys. Since Treedys is not specifically designed for head-scale capture, the authors apply post-process to crop the head meshes based on 3D landmarks, and removing the rest part of the upper body. HUMBI [54] is a large-scale dataset, which contains different body part collections. As the systems for these two datasets are not customized to best fit head-level capture, they are limited in resolution. Multiface [51] contains head-oriented collections and detailed annotations, but only releases 13 subjects' data. To facilitate multisensory modeling, VOCASET [11] is proposed. It is a 4D speech-driven scan dataset with about 29 minutes of 4D scans and synchronized audio from 12 speakers. Although VOCASET allows training and testing of speech-to-animation geometric models and can generalize to new data, it is limited in the extremely narrow diversity of subjects and onefold task. The other alternative is audio-visual data. These datasets are widely used in audio-visual learning tasks, like lip reading [7, 6], speaker detection [40] and talking head generation [47, 58]. For example, GRID [9] and MEAD [47] are sparse multi-view datasets (four and eight respectively), which are characterized by consistent shooting conditions, carefully designed identity, and corpus distribution. However, the sparsity leaves these datasets more often to be used in 2D methods.
In contrast, our RenderMe-360, is a large-scale multi-view dataset for high-fidelity head avatar creation research. It is under a head-oriented, and high-resolution data capture environment. It contains diverse data samples (with 500 subjects performing various activities, e.g., expressions,
Figure 2: Overview of data collection pipeline. (a) Our system, named as POLICY, records subjects' performances in 60 different views. (b) Illustration of data collection process. (c) Illustration of annotation pipeline.
speeches, and hair motions), multi-sensory data, and rich annotations. It is a comprehensive digital asset library that ensures the compatibility of evaluating multiple head avatar tasks in one single dataset. A comparison between RenderMe-360 and other related datasets is shown in Table 1.
2.2 Neural Rendering for Head Avatar
Due to space limits, we discuss methods for head rendering in main paper, and unfold the other three directions - hair reconstruction, hair editing, and talking head generation in supplementary.
Head Rendering. Head or face priors are often used to condition the neural fields. Such a philosophy can help either improve the robustness or create controllable avatars [16, 2, 5]. Priors like parametric model [4, 22] coefficients, key points, and explicit surface mesh/point clouds are popular ones to be integrated into the framework. For example, NHA [17] presents a framework to learn vertex offsets and attached textures from fitted FLAME surface via coordinate-based MLPs embedded on the surface. NerFACE [15] and IM Avatar [56] use FLAME [22] model expression coefficient to condition the neural field and learn to create an animatable head avatar from monocular video. Taking multi-view images as input condition, Neural Volume [30] models dynamic 3D content with a volumetric representation. MVP [31] replaces the single volume with a mixture of multiple predefined volumetric primitives which improves the resolution and efficiency of volume rendering. To increase the flexibility of re-rendering the avatar in new environments (e.g., novel expression and lighting), PointAvatar [57] presents a paradigm of utilizing point-based representation which achieves fast model convergence by coarse-to-fine optimization. For generalization, KeypointNeRF [32] synthesizes free viewpoints of human heads via multi-view image features and 3D keypoints. In addition, some researchers use cross-domain data such as audio or text to condition the neural fields. For instance, ADNeRF [18] presents a 3D-aware alternative to the 2D talking face pipelines by conditioning the radiance field with both head poses and audio fragments.
3 RenderMe-360
In this section, we introduce RenderMe-360 dataset in detail. We start with the description of our capture system (Section 3.1), and move on to an overview process of data collection (Section 3.2). Then, we present the data annotation pipeline (Section 3.3). The whole process is visualized in Figure 2.
3.1 Capture System
As illustrated in Figure 2(a), we build a multi-video camera system to record synchronized multi-view videos of human head performance. It contains 60 industrial cameras and covers a field of view of  $360^{\circ}$  left-to-right and over  $160^{\circ}$  up-to-down for video capture at the whole-head level. To ensure encompassing fine details (e.g., hair strands, wrinkles, and freckles), we choose cameras with a high resolution at  $2448 \times 2048$ . The shutter speed is 30 FPS for capturing fine-grained motion changes. To capture multi-sensory information, a condenser microphone is collocated with the camera system, and under the audio-vision synchronization. Please refer to supplementary for more system details.
The data collection pipeline is illustrated in Figure 2(b). Specifically, to guarantee the valid rate of captured data, we first apply a trial collection with a fake head to check on the operability of equipment and adjustment of camera positions before formal acquisition. After this, we start the formal capture process with the following parts for per-person recording: 1) Calibration Capture. We capture camera calibration data before every round of recording. We use a chessboard and move it in front of the cameras at a fixed-order trajectory. 2) Expression Capture. We ask each subject to
perform the same expression set, which includes 12 distinctive facial expressions (1 natural and 11 exaggerated expressions) defined in [53]. 3) Hair Capture. To cover diverse hair materials and hair motions, we record 12 video sequences (on average) for each normal subject, with different hairstyles under three levels—original hair, headgear, and wig captures. Specifically, the collected data includes one motion sequence for the subject's original hair, one for headgear that hides one's hair, and rest sequences for wearing different wigs with random styles and colors. 4) Speech Capture. We provide rich corpus that encompasses single words combined sentences, phonetically balanced protocols, and short paragraphs in two languages (Mandarin and English). For each subject, we randomly pick materials from the corpus and ask the subject to speak 26 or 42 phonetically balanced sentences.
We obtain a large-scale dataset of over  $800k$  recording videos from 500 identities at the end, which is gender-balanced, includes multiple ethnicities, and spans ages from 6 to 88 with approximate normal distribution where teenagers and adults form the major part (Figure 3). A more detailed description of our data collection process and related data statistics are discussed in the Supplementary Materials.
3.3 Data Annotation
Diverse and multi-granularity head-related annotations are crucial for the research of human head avatar tasks. However, there is still a deficiency of an all-around head dataset with rich annotation in the research community. To facilitate the development of downstream tasks, we provide rich annotations. We also provide a toolbox to automatically label most of the annotations (Figure 2(c)). We unfold the key information in this section. For more details, please refer to Supplementary.
Camera Parameters. We estimate extrinsic matrix and rectify intrinsic matrix for each camera via a fine checkerboard pipeline [8]. The process includes checkerboard detection, intrinsic calibration, and extrinsic calibration with multi-view bundle adjustment. To ensure quality, we additionally applied fast novel view synthesis [34], and facial landmark reprojection on multi-view single-frame to eliminate unqualified estimation on camera parameters.
2D & 3D Facial Landmarks. 2D landmarks are per-frame detected via an enhanced version of [50] on selected frontal views, which range from  $60^{\circ}$  left to  $60^{\circ}$  right. With 2D landmarks from multiple views, RANSAC [14] triangulation is applied to obtain 3D landmarks. To guarantee accuracy, low-quality 2D ones are filtered out with spatial and temporal constraints, and 3D results with large re-projection errors are also
Figure 3: Key data statistics. a) 26 or 42 speeches are recorded per subject. b) Annotation example. c) 12 expressions are captured for each subject. d) 10 wigs (on average) are randomly sampled for each normal subject. Original hair and headgear are also captured. e) Distribution of gender, age, and ethnicity. Better zoom in for details.
filtered out. For the frames that are neither precisely calculated on 2D landmarks nor 3D landmarks, we manually label the 2D landmarks and re-run the triangulation.
**Dense Mesh Reconstruction.** Traditional MVS algorithms based on feature points extraction and geometric optimization, such as [46], can only generate irregular point clouds, and have low-quality results in areas of texture missing, such as black hair and dark skin. Therefore, we additionally apply NeuS [48], which uses neural representation for signed-distance-function and optimizes with surface-based rendering results to do multi-view reconstruction and dense mesh extraction. For video sequences, the first frame is optimized from scratch, then the following frames are fine-tuned on the optimized neural representation to accelerate convergence speed.
Matting. Reasonable foreground segmentation for human heads is challenging. Since diverse hairstyles and accessories form the long-tail problem. Thus, we develop a united pipeline that combines video-based matting and scan mesh information to improve matting quality. Specifically, we capture the background prior to each round of recordings, and apply RVM [25] to estimate the rough matting result in the first step. We additionally blend depth-aware mask via Z-buffer during rasterization [26] on the scanned mesh to improve matting quality. With multi-view information, the background ambiguity can be distinguished from other views. We use Gaussian Mixture Model to blend the estimations from two models for each pixel[41]. For extremely hard cases where both steps cannot output satisfactory results, we add human-in-the-loop labeling.
FLAME Fitting. Since only keyframes are attached with scan meshes to save processing costs, we use two fitting methods in practice - one is fitted with scan mesh, and the other is not. For frames with
scans, we use 3D landmarks to initialize FLAME parameters and optimize via ICP [1]. Since scan shows accurate facial shape in world space, fitting with it preserves better facial contour. For the ones without scans, personalization is designed. The key is to select neutral frames with corresponding scan geometries, and average across these fitting results to generate a personalized template for each subject. We keep shape parameters constant during fitting the expression sequences which lack scans.
Text Annotation. To facilitate multi-modal research on human head avatars, we provide text descriptions for the captured videos at unprecedented granularity. These descriptions cover both static and dynamic attributes from four major aspects: 1) static facial features of the subjects, where over 90 attributes at general facial appearance, detailed appearance, and lighting condition levels are described; 2) static information of non-facial regions, where the texture, material, and shape attributes of subject's accessories (such as necklace, earrings, and hairpin) and hairstyle are defined; 3) dynamic facial actions, fine-grained action units (AUs) descriptions based on the FACs system[12] are given; 4) dynamic video activity descriptions, where full-sentence annotations of global action sequence descriptions for each captured video are provided.
Our RenderMe-360 dataset provides various potentials in new research directions and applications for human avatar creation. Here, we build a comprehensive benchmark upon RenderMe-360 dataset, with 16 representative methods on five vital tasks of human head avatars (summarized in Table 2). These tasks range from static head reconstruction and dynamic synthesis to generation and editing. For each task, we set up several experimental protocols to probe the performance limits of current state-of-the-art methods under different settings. Due to space limits, we present the key insights of the novel view synthesis task in the main paper, and highlight the best/worst results in lavender/gray colors. Please refer to Supplementary Materials for benchmarks of the rest four tasks (novel expression synthesis, hair rendering, hair editing, and talking head generation), more implementation details, experiments, qualitative/quantitative results, and discussions.
4.1 Novel View Synthesis
Here, we present novel view synthesis (NVS) benchmark of both case-specific (i.e., Single-ID NVS in sub-section 4.1.1) and generalizable (sub-section 4.1.2) tracks.
4.1.1 Single-ID NVS
This case-specific track refers to the setting of training on a single head with multi-view images, which originates from NeRF [33]'s de facto setting, to evaluate the robustness of static multi-view head reconstruction. We study four representative methods with two protocols - 1)#Protocol-1 for exploring methods' robustness to different appearance or geometry factors. The dataset is split into three categories, i.e., Normal Case, With Deformable Accessory, and With Complex Accessory, according to the complexity of appearance and geometry. We discuss the protocol in the main paper; 2)#Protocol-2 for probing methods' robustness to different camera number and distributions. We discuss this protocol in Supplementary Materials.