SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Youngjoong Kwon $^{1,*}$ , Lingjie Liu $^{2,3}$ , Henry Fuchs $^{1}$ , Marc Habermann $^{3,4†}$ , Christian Theobalt $^{3,4}$
<sup>1</sup>University of North Carolina at Chapel Hill. <sup>2</sup>University of Pennsylvania.
3Max Planck Institute for Informatics, Saarland Informatics Campus.
$^{4}$ Saarbrücken Research Center for Visual Computing, Interaction and AI. {youngjoong, fuchs}@cs.unc.edu {lliu, mhaberma, theobalt}mpi-inf.mpg.de
Generating controllable and photorealistic digital human avatars is a long-standing and important problem in Vision and Graphics. Recent methods have shown great progress in terms of either photorealism or inference speed while the combination of the two desired properties still remains unsolved. To this end, we propose a novel method, called DELIFFAS, which parameterizes the appearance of the human as a surface light field that is attached to a controllable and deforming human mesh model. At the core, we represent the light field around the human with a deformable two-surface parameterization, which enables fast and accurate inference of the human appearance. This allows perceptual supervision on the full image compared to previous approaches that could only supervise individual pixels or small patches due to their slow runtime. Our carefully designed human representation and supervision strategy leads to state-of-the-art synthesis results and inference time. The video results and code are available at https://vcai.mpi-inf.mpg.de/projects/DELIFFAS.
Generating photorealistic renderings of humans is a long-standing and important problem in Computer Graphics and Vision with many applications in the movie industry, gaming, and AR/VR. Traditionally, creating such digital avatars from real data involves complicated hardware setups and manual intervention from experienced artists, followed by sophisticated physically-based rendering techniques to render them into an image. Thus, recent research and also this work focuses on creating a drivable and photoreal digital double of a real human learned solely from multi-view video data to circumvent the complicated and manual work.
Recent works can be categorized by their underlying scene representation. Explicit mesh-based methods represent the dynamic human by a deforming geometry and dynamic textures [15] or textured body models [45]. While these explicit methods achieve a comparably fast inference speed, the quality is still limited in terms of detail and photorealism. Hybrid approaches [14, 32] integrate the coordinate-based MLP to an explicit (potentially deforming) geometry. While their synthesis quality is drastically improved compared to explicit methods, the runtime is rather slow such that real-time is out of reach. Last, the concept of light fields [13, 38, 5] is well known for decades and recently, neural variants [47, 30, 55, 2, 52] have been proposed. Since ray-marching, i.e., sampling multiple points along the ray, is not required, these methods are fast to evaluate. However, results are
*This work was completed during an internship at MPII.
Figure 1: We present DELIFFAS, a novel method for real-time, controllable, and highly-detailed human rendering. Our method takes a skeletal motion and camera view as input and generates a corresponding photorealistic image of the actor in real-time.
only shown on static scenes. Thus, previous work either demonstrated high synthesis quality or fast inference speed. Our goal is to have a method, which achieves the best of the two worlds, i.e., high synthesis quality and fast inference speed.
To this end, we propose DELIFFAS, a method for controllable, photorealistic, and fast neural human rendering. Given a multi-view video of an actor and a corresponding deformable human mesh model, which computes the deformed mesh as a function of the skeletal motion, we propose a deformable light field parameterization around the template. Typically, a light field is parameterized by a 3D position and a direction, however, this parameterization is neither robust nor efficient for deforming scenes. Therefore, we propose a deformable two-surface representation parameterizing a surface light field. In contrast to the original surface light field formulation, we deform the two surfaces according to a deformable mesh model. This allows us to efficiently query highly detailed appearance and even enables controllability since the light field can be driven by the skeletal motion. Due to the high inference speed, our method is also able to render the entire image during training, which is in stark contrast to coordinate-based approaches [32]. This allows us to employ perceptual supervision on the entire image, which is in contrast to previous works that could only supervise individual pixels or small patches due to their slow runtime. In summary, our technical contributions are:
- A novel real-time method for learning controllable and photorealistic avatars efficiently from multi-view RGB videos.
- A deformable two-surface representation parameterizing a surface light field, which allows efficient and highly accurate appearance synthesis.
- We show that our surface light field representation and our neural architecture design can be effectively parallelized and integrated into the graphics pipeline enabling real-time performance and also allowing us to employ full-image level perceptual supervision.
Explicit Mesh-based Methods. Some methods use an explicit mesh template and estimate non-rigid deformations and textures on the template mesh to fit the RGB input [60, 61]. Xu et al. [62] use texture retrieval and stitching to synthesize a texture map for the novel view. Others [7, 54] propose a layered texture representation to accelerate the image synthesis. These methods present limited generalization to new poses and viewpoints. Thies et al. [53] learn deep features in the texture space. Although they can represent view-dependent effects, it cannot generalize to novel poses. To address this issue, DDC [15] leverages differentiable rendering techniques to learn non-rigid deformations and texture maps on the explicit mesh. Given novel poses and views, DDC [15] can synthesize plausible results in real time. However, our method demonstrates significantly improved visual quality while also being real-time capable. Chen et al. [9] is a single surface-based method that is tightly bound to the underlying mesh. In contrast, the proposed two-surface light field can compensate for potentially erroneous geometry and synthesizes photoreal appearance beyond the mesh bounds.
Hybrid Methods. The synthesis quality of explicit meshed-based methods is bounded by the limited resolution of the template mesh. On the other hand, neural implicit representations [49,
Figure 2: Method Overview. Given the skeletal motion and the target camera pose, we synthesize highly detailed appearance of a subject in real-time using a surface light field parameterized by our two surface representation. We first obtain the inner surface using the motion-dependent deformable human model. The outer surface is constructed by offsetting the inner surface vertie along its normal. For each camera ray, we obtain the uv coordinates of the intersecting points with the two surfaces from the image-space uv maps. Then, we bilinearly sample the features at the corresponding uv coordinates from the temporal normal feature map of each mesh. The two sampled features together with the two intersecting uv coordinates are fed into the light field MLP, which generates the color value. The rendered 2D image is supervised with  $L_{1}$  and perceptual losses.
33, 24, 37, 31, 48] have shown their advantages over explicit representations in 3D modeling. For example, neural implicit representations are continuous, compact, and can achieve high spatial resolution. However, neural implicit representations are not animatable as deformable meshes. Therefore, recent studies propose to combine neural implicit representations with an explicit human template. Peng et al. [43] use a coarse deformable model (SMPL [34]) as a 3D proxy to optimize feature vectors, but it has limited pose generalizability. To address this issue, a large body of work [32, 41, 8, 63, 51, 39, 20, 26, 11, 66, 56, 57, 29, 50, 42, 21, 64, 27] propose neural animatable implicit representations. These works leverage the animatable 3D proxy to deform the space of different poses to a shared canonical pose space. HDHumans [14], jointly optimizes the neural implicit representation and the explicit mesh model. However, these methods suffer from slow rendering speed, taking about 5 seconds to render one frame. In contrast, our method can achieve real-time rendering (31fps) and produces higher-quality synthesis.
Light-field Methods. The 5D plenoptic function, which represents the intensity of light observed from every position and direction in 3-dimensional space, was introduced by Bergen and Adelson [5]. Levoy et al. [28] reduce the light field to a 4D function with the two-plane parameterization, i.e., light slab. Wood et al. [58] propose a surface light field, which maps a point on the base mesh and viewing direction to the radiance. While they can parameterize a full 360-degree ray space, they cannot go beyond the underlying mesh and can only represent a static scene. Recently, learning-based light field approaches have been proposed. Convolutional neural network-based methods [23, 4, 36] learn to interpolate and extrapolate the light field from a sparse set of views. Some works [30, 55, 2] achieve high quality synthesis with the light field represented as an MLP. However, they are limited to the fronto-parallel scene synthesis. To represent the full 360-degree ray space, some works [47, 52] utilize Plücker and two-sphere parameterization [18, 6], respectively. However, none of the aforementioned works can represent the full  $360^{\circ}$  light space of a dynamic scene. Ouyang et al. [40] leverages multiplane image and perceptual supervision to render a photorealistic avatar in real-time. However, they only allow minimal pose changes. In our work, we introduce a deformable two-surface representation parameterizing the surface light field, which enables us to represent and control the dynamic scene, i.e., the human, and to handle arbitrary poses during inference.
Our goal is to learn an animatable 3D human character from multi-view videos, which can be rendered photorealistically from novel views and in novel motions in real time. In this endeavor, we propose a novel method, called DELIFFAS, which at inference takes a skeletal motion sequence and a target
camera pose as input and renders a high-quality image of the subject in the specified pose under the virtual camera view at real-time framereates. An overview of our approach is shown in Fig. 2. At the core, our method represents the digital human as a surface light field, which is parameterized by two deforming surfaces that are directly controlled through the skeletal motion. Next, we provide some background concerning our data assumptions, deformable surface representations, and light fields (Sec. 3.1). Then, we introduce our deformable surface light field parameterized by a motion-dependent deformable two-surface representation and how it can be efficiently rendered (Sec. 3.2). Lastly, we introduce our perception-based supervision strategy (Sec. 3.3).
Data Assumptions. We assume a segmented multi-view video of the actor, who is performing a diverse set of motions, is given using  $C$  calibrated and synchronized cameras. Moreover, we also assume the skeletal motion  $\pmb{\theta}_f \in \mathbb{R}^D$  for a frame  $f$  of the video can be obtained, e.g., by using markerless motion capture. Here,  $D$  denotes the number of degrees of freedom of the skeleton.
Skeletal Motion-dependent Surface. In this work, we mainly focus on the question of how to efficiently render a person at a photorealistic quality and less on surface reconstruction or modeling. Thus, we assume a skeletal motion-dependent deformable mesh surface  $s(\theta_{f,f - T}) = \mathbf{V}_{\theta_f}$ :  $\mathbb{R}^{T\times D}\to \mathbb{R}^{N\times 3}$  of the human is given. Here,  $\theta_{f,f - T}$  denotes the motion window from  $f - T$  to the current pose  $f$  and  $\mathbf{V}_{\theta_f}$  denotes the deformed and posed mesh vertex positions at frame  $f$  with pose  $\theta_{f}$ .  $N$  is the number of vertices. In practice, we leverage the deformable human model of Habermann et al. [16]. However, any other deformable surface representation such as SMPL [34] could be used as well (see Sec. D for more discussions).
Light Field. The light field [13, 38, 5] is a function  $l(\mathbf{x}, \mathbf{d}) = \mathbf{c}$  that maps an oriented camera ray, here parameterized by its origin  $\mathbf{x} \in \mathbb{R}^3$  and the direction  $\mathbf{d} \in \mathbb{R}^3$ , to the transmitted outgoing radiance along the camera ray. Note that only a single evaluation per pixel is required when rendering the light field into a discrete image, which is in stark contrast to the recently proposed Neural Radiance Fields [37] that require hundreds of computationally expensive network evaluations along the ray. Thus, light fields offer a computationally efficient alternative. Among the many possible ways of parameterizing a ray, the two-plane parameterization [28], i.e., light slab, is the most commonly used method. This two-plane method parameterizes a ray by its intersections with two planes  $\mathcal{U}_1$  and  $\mathcal{U}_2 \in [0,1]^2$ . Without loss of generality, we here define the planes being contained in a range from zero to one and assume the field is always first intersecting  $\mathcal{U}_1$  and then  $\mathcal{U}_2$ . Thus, the light field formulation can be adapted to
$$
l \left(\mathbf {u} _ {1}, \mathbf {u} _ {2}\right) = \mathbf {c} \tag {1}
$$
where  $\mathbf{u}_1 \in \mathcal{U}_1$  and  $\mathbf{u}_2 \in \mathcal{U}_2$  are the intersection points of a ray with the planes. However, it is hard to model full  $360^\circ$  view changes with this light slab representation, and it originally was not designed for controlling the scene, in our case the human. Thus, in the remainder of this section we demonstrate how this classical concept can be extended to our setting and how our formulation enables fast and photoreal renderings of humans.
3.2 Deformable Light Field Parameterization
As discussed above, one major challenge for the traditional two-plane representation is that it cannot cover the full  $360^{\circ}$  in terms of viewing directions. Thus, we first explain how we adopt this representation using the deformed character mesh.
$360^{\circ}$  Viewpoints. Since we have the deformed mesh  $\mathbf{V}_{\theta_f}$ , our idea is to adapt the two-plane parameterization to a two-(non-planar)-surface parameterization using the deformed mesh. While the inner surface (equivalent to  $\mathcal{U}_2$ ) can be represented by the original mesh  $\mathbf{V}_{\theta_f}$  (denoted as  $\mathbf{V}_{\theta_f}^{\mathrm{inner}}$  in the remainder), we have to construct an outer surface  $S_1$  for which we can guarantee that every ray first intersects  $S_1$  before intersecting  $S_2$ . This can be achieved by constructing an offset surface as
$$
\mathbf {V} _ {\boldsymbol {\theta} _ {f}, i} ^ {\text {o u t}} = \mathbf {V} _ {\boldsymbol {\theta} _ {f}, i} ^ {\text {i n n e r}} + d \cdot \mathbf {n} _ {i} ^ {\text {i n n e r}} \tag {2}
$$
where  $i$  denotes the  $i$ -th vertex,  $\mathbf{n}_i^{\mathrm{inner}}$  is the respective normal on the inner surface, and  $d$  defines the length of the offset. We use  $\mathrm{d} = 3\mathrm{cm}$  in the experiments. By offsetting all vertices, we obtain the outer surface  $\mathbf{V}_{\theta_f}^{\mathrm{out}}$ . By assuming the mesh is watertight and that the origin  $\mathbf{o}$  of a ray with direction  $\mathbf{d}$  lies
outside of  $\mathbf{V}_{\theta_f}^{\mathrm{out}}$ , we can guarantee that this ray will always intersect  $\mathbf{V}_{\theta_f}^{\mathrm{out}}$  before  $\mathbf{V}_{\theta_f}^{\mathrm{inner}}$ . Note that we can now have plausible intersections with the two surfaces from (almost) any point in 3D space enabling  $360^{\circ}$  viewpoints, which are typically difficult for the classical two-plane parameterization.
2D Surface Parameterization. Now if the ray intersects the outer and inner surface at  $\mathbf{p}^{\mathrm{out}}$  and  $\mathbf{p}_{\theta_f}$  , respectively, we are interested in converting it into the original light slab parameterization (Eq. 1) that solely consists of two 2D coordinates. We achieve this by performing UV mapping, i.e., we construct a texture atlas  $m(\mathbf{p}) = \mathbf{w}$  for the original mesh  $\mathbf{V}_{\theta_f}^{\mathrm{inner}}$  .  $m$  maps a point  $\mathbf{p}$  on the 3D surface to a 2D location  $\mathbf{w}\in [0,1]^2$  on a plane. Note that the atlas remains the same across poses since the connectivity of the mesh is fixed. Further, our offset surface construction also preserves the texture atlas and, thus, the atlas for the outer surface is equivalent to the inner one. Now, our 3D intersection points  $\mathbf{p}^{\mathrm{out}}$  and  $\mathbf{p}_{\theta_f}^{\mathrm{inner}}$  can be mapped to 2D as  $m(\mathbf{p}^{\mathrm{out}}) = \mathbf{w}^{\mathrm{out}}$ $m(\mathbf{p}_{\theta_f}^{\mathrm{inner}}) = \mathbf{w}_{\theta_f}^{\mathrm{inner}}$  and the light slab formulation can be adapted as
$$
l \left(\mathbf {w} ^ {\text {o u t}}, \mathbf {w} ^ {\text {i n n e r}}\right) = \mathbf {c}. \tag {3}
$$
Note that both 2D coordinates are not dependent on the motion of the character and, therefore, the light field  $l$  cannot model motion-dependent appearance. Next, we further refine our formulation such that  $l$  can potentially model this.
Skeletal Motion-dependent Conditioning. We note that the human mesh  $\mathbf{V}_{\theta_f}^{\mathrm{inner}}$  is already a function of skeletal motion, though it is not in a format for efficient encoding considering the light slab formulation. We convert the inner and outer surface into temporal normal maps [15]  $\mathcal{T}_{\theta ,f:f - T}^{\mathrm{inner}}$  and  $\mathcal{T}_{\theta ,f:f - T}^{\mathrm{out}}$ , which are the concatenation of the posed normal maps of the motion window  $[f,f - T]$ . Note that both maps encode information about the skeletal motion and can be directly generated by the motion-dependent mesh surface  $s(\cdot)$ .
Now, we deeply encode them into feature maps  $f_{\Omega}(\mathcal{T}_{\theta ,f:f - T}^{\mathrm{inner}})$  and  $f_{\Psi}(\mathcal{T}_{\theta ,f:f - T}^{\mathrm{out}})$  with a channel size of 32 respectively using deep convolutional networks  $f$  with weights  $\Omega$  and  $\Psi$ . Last, our light slab formulation (Eq. 3) can be again refined as
$$
l (\gamma (\mathbf {w} ^ {\mathrm {o u t}}), \gamma (\mathbf {w} ^ {\mathrm {i n n e r}}), \Pi (f _ {\Psi} (\mathcal {T} _ {\theta , f: f - T} ^ {\mathrm {o u t}}), \mathbf {w} ^ {\mathrm {o u t}}), \Pi (f _ {\Omega} (\mathcal {T} _ {\theta , f: f - T} ^ {\mathrm {i n n e r}}), \mathbf {w} ^ {\mathrm {i n n e r}})) = \mathbf {c}. \qquad (4)
$$
Here,  $\Pi$  denotes the bilinear sampling operator.  $\gamma$  is the positional encoding [37]. Intuitively, the light slab  $l$  takes the bilinearly sampled features on the UV plane at the uv coordinates of the intersections points of the ray with the respective plane. Now, our light slab formulation is not only view-dependent but is also motion-dependent since the features are encoding the skeletal motion.
In case the ray is not intersecting the inner surface, inspired by depth peeling, uv coordinates and features are sampled at the second intersection with the outer surface. Here, the watertightness of the model guarantees that such a second intersection point exists. We note that our light slab  $l$  is represented as an 8-layer MLP with 256 channel-size.
Efficient Rendering. Next, we explain how the above formulation can be efficiently computed using the standard graphics pipeline and deep learning tools. First, the two convolutional networks  $f_{\Psi}$  and  $f_{\Omega}$  have to encode the respective normal maps. The computational effort is independent of the number of foreground pixels, i.e., pixels which are covered by the subject. Next, for each pixel we have to obtain the uv coordinates of the intersection points with the inner and outer surface. Here, standard GPU-parallelized rasterization can be leveraged by rendering the uv coordinates into screen space. The only computation that linearly scales with the number of foreground pixels is the evaluation of the light slab  $l$ , but since there is only one evaluation per ray, also this step is computationally efficient leading to an inference time of 31 fps.
3.3 Supervision and Training Procedure
We supervise our approach by minimizing the following loss  $\mathcal{L} = \lambda_1\cdot \mathcal{L}_1 + \lambda_{\mathrm{perc}}\cdot \mathcal{L}_{\mathrm{perc}}$  , where  $\mathcal{L}_1$  and  $\mathcal{L}_{\mathrm{perc}}$  are the  $L_{1}$  loss and perceptual [22] loss, respectively.  $\lambda_{1}$  and  $\lambda_{\mathrm{perc}}$  are their weights. We use the VGG-16 network [46] pre-trained on ImageNet [10] to compute the perceptual similarity loss. Although our method solely supervised with the  $L_{1}$  loss already outperforms other real-time methods and most of the non-real time methods on the novel view synthesis task (see Tab. 1a and Tab. 2-c), it is still blurry. This is due to the one-to-many mapping problem [3, 32] where the single skeletal pose can still induce various different appearances. Thus, the perceptual loss [22] is additionally employed
Method	PSNR↑	LPIPS↓	FID↓	Real-time
NB	29.94	42.15	109.98	X
A-NeRF	29.54	35.27	86.90	X
NA	30.21	23.60	18.56	X
HDHumans	31.00	14.61	4.93	X
NHR	28.39	46.07	116.59	✓
NV	25.49	85.69	123.19	✓
DDC	32.96	20.07	27.73	✓
Ours	33.30	12.85	8.69	✓
(a) View synthesis error on  $D2$
Method	PSNR↑	LPIPS↓	FID↓	Real-time
NB	29.37	43.99	115.70	X
A-NeRF	28.42	38.74	95.56	X
NA	28.43	28.33	24.50	X
HDHumans	27.69	24.00	9.25	X
NHR	28.08	47.65	122.60	✓
NV	23.32	98.35	139.82	✓
DDC	28.05	30.43	38.37	✓
Ours	27.95	26.59	26.16	✓
(b) Motion synthesis error on  $D2$
Table 1: Quantitative results on novel view and pose synthesis. (a) Our method achieves the best performance among all the real-time and non-real time methods in terms of PSNR and LPIPS, and the second best performance on FID. (b) Our method again achieves high performance on the perceptual metrics while running in real-time. While HDHumans has slightly better results in terms LPIPS and FID compared to our work, their method is  $\times 100$  slower than our method.
and this improves the fine-grained details as can be seen in the ablation study (see Fig. 5-c,d). In contrast to patch-based perceptual supervision methods [44, 35, 57, 12], our fast rendering speed enables employing the perceptual supervision on the entire image.
DELIFFAS runs at 31fps when rendering a  $1K$  ( $940 \times 1285$ ) video using a single A-100 GPU with an Intel Xeon CPU. The implementation details and additional results are provided in the appendix.
Dataset. We evaluate our method on the DynaCap dataset [16], which is publicly available. DynaCap provides performance videos of 5 different subjects captured from 50 to 101 cameras, foreground mask, and skeletal pose corresponding to each frame, and the template mesh for each subject. The training and testing videos consist of around  $20k$  and  $5k$  frames at the resolution of  $1K$  ( $1285 \times 940$ ), respectively. Four cameras are held out for the testing and the remaining ones are used for the training as proposed by the original dataset. Among the available subjects, we choose  $D_{1}$ ,  $D_{2}$ , and  $D_{5}$  subjects for our experiments, which vary in terms of clothing style, i.e., loose skirts and trousers.