DELIFFAS: Deformable Light Fields for Fast Avatar Synthesis
Authors: YoungJoong Kwon
Lingjie Liu
Henry Fuchs
Marc Habermann
Christian Theobalt
Venue: NeurIPS 2023 poster (2023)
Keywords: DELIFFAS: Avatar Modeling
Avatar Synthesis
Animatable Human
Light Fields
Human Performance Capture

Abstract: Generating controllable and photorealistic digital human avatars is a long-standing and important problem in Vision and Graphics. Recent methods have shown great progress in terms of either photorealism or inference speed while the combination of the two desired properties still remains unsolved. To this end, we propose a novel method, called DELIFFAS, which parameterizes the appearance of the human as a surface light field that is attached to a controllable and deforming human mesh model. At the core, we represent the light field around the human with a deformable two-surface parameterization, which enables fast and accurate inference of the human appearance. This allows perceptual supervision on the full image compared to previous approaches that could only supervise individual pixels or small patches due to their slow runtime. Our carefully designed human representation and supervision strategy leads to state-of-the-art synthesis results and inference time. The video results and code are available at https://vcai.mpi-inf.mpg.de/projects/DELIFFAS.

DELIFFAS: Deformable Light Fields for Fast Avatar Synthesis
Youngjoong Kwon'; Lingjie Liu Henry Fuchs',
Mare Habermann Christian Theobalt
University of North Carolina at Chapel Hill. University of Pennsylvania. Max Planck Institute for Informatics, Saarland Informatics Campus.
Saarbrüückn Research Center for Visual Computing, Interaction and AI.
(youngjoong oong, fuchs) @cs. @cs. unc. edu (1liu, mhaberma, tteothlt haplltmmpiiii .pg de
Abstract
Generating controllable and photorealistic digital human avatars is a long-standing and important problem in Vision and Graphics. Recent methods have shown great progress in terms of either photorealism or inference speed while the combination of the two desired properties still remains unsolved. To this end, we propose a novel method, called DELIFFAS, which parameterizes the appearance of the human as surface light field that is attached a controllable and deforming human mesh model. At the core, represent the light field around the human with a deformable two-surface parameterization, which enables fast and accurate inference of the human appearance. This allows perceptual supervision on the full image compared to previous approaches that could only supervise individual pixels or small patches due to their slow runtime. Our carefully designed human representation and supervision strategy leads to state-of-the-as synthesis results and inference time. The video results and code available at htttps://vcai mpi- inf wwg.ppg...... wwwp///oooocssDEELFFP.P.
Introduction
Generating photorealistic renderings of humans a long-standing and important problem in Computer Graphics and Vision with many applications in the movie industry, gaming, and AR/VR. Traditionally, creating such digital avatars from real data involves complicated hardware setups and manual intervention from experienced artists, followed by sophisticated physically-basee rendering techniques to render them into an image. Thus, recent research and also this work focuses on creating a drivable and photoreal digital double of real human learned solely from multi-view video data to circumvent the complicated and manual work.
Recent works can be categorized by their underlying scene representation. Explicit mesh-basd based methods represent the dynamic human by a deforming geometry and dynamic textures [S5] or textured body models 45 While these explicit methods achieve comparably fast inference speed, quality is still limited in terms detail and photorealism. Hybrid approaches [14 32 integrate the coordinate-based MI.P to an explicit (potentially deforming) geometry. While their synthesis quality drastically improved compared to explicit methods, the runtime is rather slow such that real-time is out of reach. Last, the concept of light fields [13 38 (5) is well known for decades and recently, neural variants [47 30 [55] [52] have been proposed. Since ray-marching, i.e., sampling multiple points along the ray, is not required, these methods are fast evaluate. However, results
"This work was completed during an internship at MPII. Cooresppnndig author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

motion
DELIFFAS
Real-time
synthesis
Camern pose
Novel views
Novel motions
Figure 1: We present DELIFFAS, novel method for real-time, controllable, and highly- detaiied human rendering. Our method takes a skeletal motion and camera view as input and generates a corresponding photorealistic image of the actor in real-time.
only shown static scenes. Thus, previous work either demonstrated high synthesis quality fast inference speed. Our goal is to have method, which achieves the best of the two worlds, ie., high symthesis quality and fast inference speed.
To this end, we propose DELIFFAS, a method for controllable, photorealistic, and fast neural human rendering. Given multi-view video of an actor and corresponding deformable human mesh model, which computes the deformed mesh as function of the skeletal motion, we propose a deformable light field parameterization around the template. Typically, a light field is parameterized by 3D position and a direction, however, this parameterization neither robust nor efficient for deforming scenes. Therefore, we propose deformable two-surface representation parameterizing surface light field. In contrast to the original surface light field formulation, we deform the two surfaces according to deformable mesh model. This allows to efficiently query highly detailed appearance and even enables controllability since the light field can be driven by the skeletal motion. Due to the high inference speed, our method is also able render the entire image during training, which is in stark contrast to coordinate-based approaches [32]. This allows us to employ perceptual supervision the entire image, which is in contrast to previous works that could only supervise individual pixels or small patches due to their slow runtime. In summary, our technical contributions are:
A novel real-time method for learning controllable and photorealistic avatars efficiently from multi-view RGB videos.
A deformable two-surface representation parameterizing a surface light field, which allows efficient and highly accurate appearance synthesis.
We show that our surface light field representation and our neural architecture design can be effectively parallelized and integrated into the graphics pipeline enabling real-time performance and also allowing us to employ full-image level perceptual supervision.
Related Work
Explicit Mesh-based Methods. Some methods use an explicit mesh template and estimate non-rigid deformations and textures the template mesh to fit the RGB input [60) 61]. Xu et al. 62 use
propose layered texture representation to accelerate the image synthesis. These methods present limited generalization to new poses and viewpoints. Thics et al. [53] Iearn deep features in the texture space. Although they represent view-dependent effects, cannot generalize to novel poses. To address this issue, DDC [5] leverages differentiable rendering techniques to learn non-rigid deformations and texture maps on the explicit mesh. Given novel poses and views, DDC [15] can synthesize plausible results in real time. However, our method demonstrates significantly improved visual quality while also being real-time capable. Chen et al. [9] is single surface- based method that is tightly bound to the underlying mesh. contrast, the proposed two-surface light field can compensate for potentially erroneous geometry and synthesizes photoreal appearance beyond the mesh bounds.
texture retrieval and stitching to synthesize a texture map the novel view. Others
Hybrid Methods. The synthesis quality of explicit meshed- methods is bounded by the limited resolution of the template mesh. On the other hand, neural implicit representations [49

---

Poodictias
Temperl Zemaal clo?
UV saap
becaus
Dedcrrate
Laaor
MLP
Colar
porsecioss
Canor
Defermabl:
surlecs
Goouad turb
Given the skeletal motion and the target camera pose, we synthesize highly detailed appearance subject in real-time using a surface light field parameterized by our two surface representation. We first obtain the inner surface using the motion-dependent deformable human model. The outer surface is constructed by offsetting the inner surface vertice along its normal. For each camera ray, obtain the coordinates of the intersecting points with the two surfaces from the image-space uv maps. Then, we bilinearly sample the features at the corresponding uv coordinates from the temporal normal feature map of each mesh. The two sampled features together with the two intersecting uv coordinates are fed into the light field MLP, which generates the color value. The rendered 2D image is supervised with L1 and perceptual losses.
Figure 2: Method Overview.
33 24 37 31 48] have shown their advantages over explicit representations in 3D modeling. For example, neural implicit representations continuous, compact, and can achieve high spatial resolution. However, neural implicit representations are not animatable as deformable meshes. Therefore, recent studies propose to combine neural implicit representations with an explicit human template. Peng et al. [43] use a coarse deformable model (SMPL (34)) as 3D proxy to optimize feature vectors, but it has limited pose generalizability. To address this issue, large body of work [32 41 & 63, 51
39 20
42 21 64 27) propose neural animatable 26 11 66 56 57 29, 550
implicit representations. These works leverage the animatable 3D proxy deform the space of different poses to shared canonical pose space. HDHumans [14), jointly optimizes the neural implicit representation and the explicit mesh model. However, these methods suffer from slow rendering speed, taking about 5 seconds to render one frame. In contrast, our method can achieve real-time rendering fp) and produces higher-quality synthesis.
Light-field Methods. The 5D plenoptic function, which represents the intensity of light observed from every position and direction in 3-dimensional space. introduced by Bergen and Adelson [5]. Levoy et al. (28) reduce the light field to a 4D function with the two-plane parameterization, i.e., light slab. Wood al. [58] propose a surface light field, which maps point on the base mesh and viewing direction to the radiance. While they can parameterize a full 360-degree ray space, they cannot go beyond the underlying mesh and can only represent static scene. Recently, learning-based light field approaches have been proposed. Convolutional neural network-based methods (23 4 36 learn to interpolate and extrapolate the light field from sparse set of views. Some works 30
55 2 achieve
high quality synthesis with the light field represented as an MLP. However, they are limited to the
fronto-pant ponto-ppaaallel scene synthesis. To represent the full 360-degree ray space, some works 47 52 utilize
Plücker and two-sphere parameterization [18 6.. respectively. However, none of the aforementioned works can represent the full 360 light space of dynamic scene. Ouyang al. [40] leverages multi- plane image and perceptual supervision to render a photorealistic avatar in real-time. However, they only allow minimal pose changes. our work, introduce deformable two-surface representation parameterizing the surface light field, which enables us to represent and control the dynamic scene, ie., the human, and to handle arbitrary poses during inference.
3 Method
Our goal is to learn animatable 3D human character from multi-view videos, which can be rendered photorealistically from novel views and in novel motions in real time. In this endeavor, we propose novel method, called DELIFFAS, which at inference takes a skeletal motion sequence and target
3

---

camera pose as input and renders high-quality image of the subject in the specified pose under the virtual camera view real-time framerates. An overview of our upproach is shown in Fig. 2 At the core, our method represents the digital human as a surface light field, which is parameterized by two deforming surfaces that are directly controlled through the skeletal motion. Next, we provide some background concerning our data assumptions, deformable surface representations, and light fields (Sec.3..1) Then, we introduce our deformable surface light field parameterized by a motion- dependent deformable two-surface representation and how it can be efficiently rendered (Sec. 3.2 Lastly, we introduce our perception- supervision strategy (Sec. 3.3
3.1 Background
Data Assumptions. We assume a segmented multi-view video of the actor, who is performing a diverse set of motions, is given using C calibrated and synchronized cameras. Moreover, we also assume the skeletal motion O I for a frame of the video can be obtained, e.g., by using markerless motion capture. Here, D denotes the number of degrees of freedom of the skeleton.
Skeletal Motion-dependent Surface. In this work. mainly focus the question of how to efficiently render a person at a photorealistic quality and less on surface reconstruction or model- ing. Thus, we assume skeletal motion-dependent deformable mesh surface f.f--))) Vo, RTxD Nx3 of the human is given. Here, f denotes the motion window from f f-T the current pose f and Ve, denotes the deformed and posed mesh vertex positions at frame f with pose 0y. NN the number of vertices. et al. [16). However, any other deformable surface representation such as SMPL [3] could be used as well (see Sec. for more discussions).
practice, we leverage the deformable human model of Habermann
is function 1(x, d) = c that maps an oriented camera ray, TR3 and the direction d E R3 the transmitted outgoing radiance along the camera ray. Note that only a single evaluation per pixel is required when rendering the light field into a discrete image, which is in stark contrast to the recently proposed Neural Radiance Fields 377 that require hundreds of computationally expensive network evaluations along the ray. Thus, light fields offer computationally efficient alternative. Among the many possible ways of parameterizing a ray, the two wwopplane parameterization (28). i.e., light slab, is the most commonly used method, This two-plane method parameterizes a ray by its intersections with two planes Ud and U2 E [o, 1ja. Without loss of generality, we here define the planes being contained in range from zero to one and assume the field is always first intersecting Un and then U2. Thus, the light field formulation can be adapted to
Light Field. The light field 13 here parameterized by its origin
38 5
I(u,, u2) =
(I)
where u, E Ui and hard to model full 360" view changes with this light slab representation, and it originally was not designed for controlling the scene, in our case the human. Thus, in the remainder of this section we demonstrate how this classical concept can be extended to our setting and how our formulation enables fast and photoreal renderings of humans.
Ue are the intersection points of ray with the planes. However, it is
3.2 Deformable Light Field Parameterization
As discussed above, one major challenge for the traditional two-plane representation is that it cannot cover the full 360 in terms of viewing directions. Thus, we first explain how we adopt this representation using the deformed character mesh.
360 Viewpoints. Since have the deformed mesh Ve,, our idea is to adapt the two-planc parameterization to wwo-(mon-planarr--in inner surface (equivalent to (42) can be represented by the original mesh Vo, (denoted as Vinner in the remainder), we have construct an outer surface S for which can guarantee that every ray first intersects Sj before intersecting Sz. This can be achieved by constructing an offset surface as
parameterization using the deformed mesh. While the
Vinnar + d n inner
0p..
where denotes the i-th vertex, juner is the respective normal on the inner surface, and d defines the length of the offset. We use d=3em in the experiments. By offsetting all vertices, we obtain the outer surface Yout, By assuming the mesh is watertight and that the origin of a ray with direction d lies

---

outside of ve, we can guarantee that this ray will always intersect Vol before Vogee, Note that we can now have plausible intersections with the two surfaces from (almost) any point in 3D space enabling 360 viewpoints, which are typically difficult for the classical two-plane parameterization.
Now if the ray intersects the outer and inner surface at p and p jiner respectively, we are interested in converting into the original light slab parameterization (Eq. that solely consists of two 2D coordinates. We achieve this by performing UV mapping, i.e., we construct texture atlas mpp) w for the original mesh Vinner, m maps point p on the 3D 0, surface to a 2D location w E [0. 1]² on a plane. Note that the atlas remains the same across poses since the connectivity of the mesh is fixed. Further, our offset surface construction also preserves the texture atlas and, thus, the atlas for the outer surface is equivalent to the inner one. Now, our 3D intersection points pour and p inner can be mapped to 2D as mpppptt) = wout m mm(piuer)) = w winerr and the light slab formulation can be adapted as
2D Surface Parameterization.
l(wour wilnerr C.
(3)
Note that both 2D coordinates are dependent on the motion of the character and, therefore, the light field I cannot model motion-dependent appearance. Next, we further refine our formulation such that can potentially model this.
Skeletal Motion-dependent Conditioning. We note that the human mesh Vinner is already a 0g
function of skeletal motion, though it is not in a format for efficient encoding considering the light slab formulation. We convert the inner and outer surface into temporal normal maps [15] Taaper and Tout f.f:f-r- which are the concatenation of the posed normal maps of the motion window [f T]. Note that both maps encode information about the skeletal motion and can be directly generated by the motion-dependen mesh surface s(-).
Now, we deeply encode them into feature maps fa (e..)-)-) and fu (. with channel size of 32 respectively using deep convolutional networks ff with weights Q and V. Last, our light slab formulation (Eq. (3) can be again refined as
((?(wo ).>))))( (w.inerr) ), IIff (To.fss-) w (( iinner T), w winner)) =c
Here, II denotes the bilinear sampling operator. y is the positional encoding (37. Intuitively, the light slab 1 takes the bilinearly sampled features on the UV plane at the coordinates of the intersections points of the ray with the respective plane. Now, our light slab formulation not only view-dependent but is also motion-dependent since the features encoding the skeletal motion.
In case the ray is not intersecting the inner surface, inspired by depth peeling, uv coordinates and features are sampled at the second intersection with the outer surface. Here, the watertightness of the model guarantees that such a second intersection point exists. We note that our light slab is represented an 8-layer MLP with 256 channel-size.
Efficient Rendering, Next, we explain how the above formulation can be efficiently computed using the standard graphics pipeline and deep learning tools. First, the two convolutional networks fo and fa have encode the respective normal maps. The computational effort is independent of the number of foreground pixels, i.e., pixels which are covered by the subject. Next, for each pixel we have to obtain the uv coordinates of the intersection points with the inner and outer surface. Here, standard PPU-paralellized rasterization can be leveraged by rendering the coordinates into screen space. The only computation that linearly scales with the number of foreground pixels is the evaluation of the light slab 1, but since there is only one evaluation per ray, also this step is computationally efficient leading to an inference time of 31fps.
3.3 Supervision and Training Procedure
We supervise our approach by minimizing the following loss L = > Li + Apere Lpere. where Li and Cpere are the L2 loss and perceptual 22 loss, respectively. A and Xpere are their weights. We use the VGG-16 network [46] pre-trained on ImageNet [10] to compute the perceptual similarity loss. Although our method solely supervised with the L1 loss already outperforms other real-time methods and most of the non-real time methods on the novel view synthesis task (see Tab. Ta and Tab. 2 it still blurry. This is due to the one-to-many mapping problem [ 32] where the single skeletal pose
can still induce various different appearances. Thus, the perceptual loss (2 is additionally employed
5

---

Method
PSNR
LPIPS]
FID.
Real-time
Method
PSNR
LPIPST
FID.
Real-time
NB
29.94
42.15
T09,98 98
K
NB
29.37
43,99
115,70
A-NeRF
29.54
35.27
86.90
X
A-NeRF
28.42
38.74
95.56
NA
30.21
23.60
18.56
NA
28.43
28.33
24.50
HDHumans
31,00
14.61
4.93
X
HDHumans
27.69
24.00
9.25
NHR
28.39
46.07
116.59
NHR
28.08
47.65
122.60
NV
25.49
85.69
123.19
23.32
98.35
139.82
DDC
32.96
20.07
27.73
DDC
28.05
30.43
38.37
Ours
33.30
12.85
8.69
Ours
27.95
26.59
26.16
(a) View synthexis error on D2
(b) Motion synthesis error on D2
Table I: Quantitative results on novel view and pose synthesis. (a) Our method achieves the best performance among all the real-time and non-real time methods in terms of PSNR and LPIPS, and the second best performance on FID. (b) Our method again achieves high performance on the perceptual metries while running real-time While HDHumans has slightly better results in terms LPIPS and FID compared to our work, their method is x 100 slower than our method.
and this improves the fine-graine grained details can be seen in the ablation study (see Fig. ( In contrast to patch-bases based perceptual supervision methods [44] 35 57 12], our fast rendering speed enables employing the perceptual supervision the entire image.
4 Results
DELIFFAS runs at 31 ps when rendering a 1 K (940 x 1285) video using a single A-100 GPU with an Intel Xeon CPU. The implementation details and additional results are provided in the appendix.
Dataset. We evaluate our method on the DynaCap dataset (16), which publicly available. DynaCap provides performance vidcos of 5 different subjects captured from 50 to 101 cameras, foreground mask, and skeletal pose corresponding to each frame, and the template mesh for each subject. The training and testing videos consist of around 20k and 5k frames the resolution of 1K (1285 x 940), respectively. Four cameras are held out for the testing and the remaining ones are used for the training proposed by the original dataset. Among the available subjects, choose D1. D2, and Ds subjects for our experiments, which vary in terms of clothing style, i.e., loose skirts and trousers.
Metrics. We evaluate our performance using the peak signal-to-noise ratio (PSNR). However, human perception cannot be fully reflected with this metric since a very blurry and unrealistic result can still lead to a low error (65). Therefore, we additionally employ the learned perceptual image patch similarity (LPIPS) (65), and the Fréchet inception distance (FID) [7], which are similar to the human perception. We generate and evaluate results at 1K resolution (1285 x 940). Every metric is computed by averaging across the entire sequence using every 10th frame. Four held-out cameras (7, 18, 27, 40), which are uniformly sampled in the space, are used for the testing.
4.1 Qualitative Results
We first present the qualitative results on the novel view synthesis task in Fig. (A) and on the novel pose synthesis task in Fig. (B). For both tasks, we show the synthesis result of the subject in two different poses viewed from two different viewpoints. Our method synthesizes images with fine-scale details including the facial features and wrinkles of the clothing for both tasks. Note that our method can even generate plausible results of the subject in a challenging garment type, i.e., loose skirt. Furthermore, we can synthesize high-quality renderings of the subject under difficult testing poses. These results confirm the versatility as well as the generalization ability of our method.
4.2 Comparison
We compare our method with the ofte--of-the-a methods concerning novel view and pose synthesis. We choose baselines from four different classes, which are volume rendering, explicit mesh hybrid, and point cloud-based methods. For the volume rendering-basee based method, we compare with Neural Volumes (NV) 333, which learns volumetric representation. Deep Dynamic Character (DDC) [16] an explicit mesh-based method, where the network learns to render the neural texture obtained by rasterizing the explicit mesh. Neural Actor (NA) [32), Neural Body (NB) 43), and A-NeRF [1 are hybrid methods that combine the human prior, i.e., naked human body model [34] and skeleton,

---

Pose
Pose 2
Posc
Pose 2
View
View
(A) Novel view synthesis
(B) Novel pose synthesss
Figure 3: Qualitative results for novel view and poses. Our method achieves photorealistic rendering quality while rumning in real time. We are able to recover the fine-grained details including the facial features and clothing wrinkles. The high-quality synthesis results on the very challenging loose garment, i..,, skirt, and the complicated testing poses show the versatility of our method.
Non-Real Time
Real-Time
(A) Novel view synthesis
NB
A-NeRF
NA
HDHumans
DDC
Ours
GT
(B) Novel puse synthesis
Figure 4: Comparisons on the novel view and pose synthesis task the D2 subject. Our method synthesizes realistic and delicate details in real time (31 fs). Note that we achieve similar or better quality offline approaches and show drastically improved quality compared to real-time methods.

---

Single-surface
b. Single-surface viewing direction
c. Two-surface
d. Two-surfaco (Ours) (L, + Lprre
GT
(4,)
Figure 5: Ablation of each design choice the novel view synthesis task. All the variants except (d) are trained only with L1 loss. Although conditioning on the viewing direction (b) improves the quality, single-surface models (a.b) are tightly bounded the underlying mesh. On the other hand, our proposed method (c,d) can go beyond the mesh and recover the real geometry. Additional utilization of the perceptual loss (d) improves the level of detail.
with NeRF Neural Actor (NA) achieves high-quality results by utilizing the adversarial supervision in texture space. Neural Human Renderer (NHR) [59] optimizes features that are anchored to point clouds. We additionally compare to the concurrent work HDHumans [14, which a hybrid method that combines the deformable human model [16] with NeRF. Similar NA, HDHumans also employs adversarial supervision in texture space. We show our comparison results on the D2 subject with tight clothing, other baselines except DDC and HDHumans cannot handle the loose garment types. Note that we present both the qualitative and quantitative results in the order of non-real time, ie., NB, A-NeRF, NA, HDHumans, and real-time methods, ie.., NHR, NV, DDC, and ours.
Fig. 4 (A) shows the comparison on the novel view synthesis task. Different from their original results 43 NB produces blurry results when trained longer and more challenging training sequences, e.g., around 20k frames, as they fail to optimize the appearance code for every frame. A-NeRF exhibits the loss of details under highly articulated poses. While DDC and NA generate coarse details, they still lack eigh-frequenc, details. On the other hand, our method can recover, both, the coarse and fine details and only achieves slightly worse synthesis quality than HDHumans while being significantly faster (x 100). In Tab. Ta our method outperforms all the non-real time and real-time methods except HDHumans. However, has a runtime in the order of seconds per frame, which prevents from being used in any real-time application. is worth noting that ours with Lı-only supervision (see Tab. () already outperforms other real-time methods and even the non-real time method NA.
would like to highlight again that HDHumans
Fig. (B) shows the comparisons the novel pose synthesis task. Again, our method can recover the sharp and
details that are close to the ones of HDHumans, while most of the other approaches struggle with the given challenging pose. This is consistent with the quantitative results Tab. [b] where we achieve the second best performance among all the methods and the best performance among the real-time methods the perception-based metrics LPIPS and FID. Due to the innate nature of pose to appearance task, i.e., the one-to-many mapping 3 (2), our method can generate realistic looking details that can deviate from the ground truth, which explains the lower performance in terms of PSNR in Tab. lb
4.3 Ablation
We perform ablation studies on the novel view synthesis task on the D2 subject. To accurately access the contribution of each component, train all the variants only with L1 loss if not stated otherwise.
Impact Two-surface Parameterization To study the efficacy of the two-surface parameteriza- tion, we train the single-surface model ("singlessuffaee) where the MLP generates color conditioned only on a single inner surface intersection coordinates and feature. We also train single-surface variant that is additionally conditioned on the viewing direction ("single-surface + viewing direction )) In Tab. 2 our two-surface model (c) outperforms the single-surface variants (a,b). Also, we would like again stress that our L Ln-oll supervised two-surface model (c) already surpasses all of the real-time methods including DDC, and also outperforms the non-real time method NA, which trained with ground truth texture-map map supervision and adversarial supervision. This confirms that the

---

Coarse mesh a. Single-surface
b. Two-surface
Original mesh c. Two-surface
GT
(Ours)
Figure 6: Ablation the deformable template quality. All baselines trained only with Lı loss. When using coarse mesh model the single-surface baseline (a) suffers from the geometric error of the model. In contrast, our two-surface representation (b) has the capability fixing geometric enrors and shows similar quality compared to the result when using the original mesh model (c).
Method
PSNR
LPIPS
FID.
a. Single-surface
31.34
22.74
28.96
b. Single-surface + viewing direction
32.19
20.38
23.88
C. Two-surface with L1
33.27
18.51
23.58
d. Two-surface with L1 + Lpere (Ours)
33.30
12.85
8.69
e. Single-surface with coarse mesh
29.63
27.75
42.84
f. Two-surface with coarse mesh
32.03
22.00
31.43
Table Ablations on novel view synthesis for the D2 subject. All baselines except (d) are trained with L1 Ln-only supervision. Our two-surface parameterization outperforms single-surface variants and the perceptual supervision further improves visual quality. Further, our results when using coarse mesh model are still plausible demonstrating the robustness of our method.
improvement originates from the carefully-designed two-surface model. In Fig. We visually ablate the impact of each design choice. Although conditioning the viewing direction (b) improves the quality, the single-surfat models (a.b) are tightly bound to the underlying mesh. On the other hand, two-surface models (c,d) go beyond the mesh and recover the real geometry.
Impact of Perceptual Supervision. Perceptual supervision improves the details (e.g., wrinkles) in Fig. d. which is also confirmed by the better performance in the perceptual metrics in Tab. d.
Impact of Deformable Template Quality. To demonstrate the robustness of method, train and evaluate our method with smoothed version of the original mesh model in Fig. and Tab. fe..f. The quantitative result outperforms real-time methods including DDC in terms PSNR. This shows that even coarser geometry is sufficient for our representation. Moreover, the single-surface model (Fig. a) reveals and suffers from geometric eттог when the coarse template is used. On the other hand, our two-surface representation (Fig. b) has the capability of fixing geometric errors and shows similar visual quality the result when the original mesh model is used (Fig. (c
5 Conclusion
We presented DELIFFAS, an animatable representation that can synthesize high-quality images of the avatar under user-controlled skeletal motion and viewpoint in real-time. At the technical core, our method utilizes a surface light field parameterized with two deformable surfaces computed from deformable human model. Then, our light slab queries pose-dependent temporal normal map features together with the intersecting uv coordinates order to render pixel color with just a single MLP evaluation per pixel. This efficient rendering allows perceptual supervision the entire image. Our experiments show that the proposed representation can synthesize high-quality renderings even under challenging poses, and outperforms the state-of oate--of-the-ar the-art real-time methods.

---

Christian Theobalt and Mare Habermann were supported by ERC Consolidator Grant 4DReply (770784). Lingjie Liu was supported by Lise Meitner Postdoctoral Fellowship. This work was partially supported by National Science Foundation Award 2107454.
Acknowledgments.
References
[1] Agisoft. Metashape. http:://ww. agisoat. com 2016.
[2] Benjamin Attal, Jia-Bin Huang. Michael Zollhöferr Johannes Kopf, and Changil Kim. Learning neural light fields with ray-space embedding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19819-19829, 2022.
[3] Timur Bagautdinov, Chenglei Wu, Tomas Simon, Fabian Prada, Takaaki Shiratori, Shih-Ea Wei, Weipeng Xu, Yaser Sheikh, and Jason Saragih. Driving-signal aware full-body avatars, ACM Transactions on Graphics (TOG), 40(4))1-17, 2021.
[4] Mojtaba Bemana. Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel. X-fields: Implicit neural view-, light-and time-image interpolation. ACM Transactions on Graphics (TOG), 39(6):1-15, 2020.
[5] James R Bergen and Edward И Adelson. The plenoptic function and the elements of early vision. Computational models of visual processing, 1:8. 1991.
[6] Emilio Camahort, Apostolos Lerios, and Donald Fussell. Uniformly sampled light fields. In Eurographics Workshop on Rendering Techniques, pages 1 117-130. Springer, 1998.
[7] Dan Casas, Marco Volino, John Collomosse, and Adrian Hilton. 4d video textures for interactive character appearance. Comput. Graph. Forum, 33(2):0, May 2014.
[8] Jianchuan Chen. Ying Zhang, Di Kang. Xuefei Zhe, Linchao Bao, and Huchuan Lu. Animatable neural radiance fields from monocular rgb video. 2021.
[9] Yue Chen, Xuan Wang, Xingyu Chen, Qi Zhang, Xiaoyu Li, Yu Guo, Jue Wang, and Fei Wang. Uv volumes for real-time rendering of editable free-view human performance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16621-16631. 2023.
[10] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 255.
leee, 2009.
[11] Qingzhe Gao, Yiming Wang. Libin Liu. Lingjie Liu, Christian Theobalt, and Baoquan Chen. Neural novel actor: Learning generalized animatable neural representation for human actors. IEEE Transactions on Visualization and Computer Graphics, 2023.
[12] Xuan Gao. Chenglai Zhong, Jun Xiang, Yang Hong, Yudong Guo, and Juyong Zhang. Reconstructing personalized semantic facial nerf models from monocular video. ACM Transactions on Graphics (TOG). 116 2022.
[13] Andreï Gershun. The light field. Journal of Mathematics and Physics, 18(144)51-151. 1939.
[14] Mare Habermann, Lingjie Liu, Weipeng Xu, Gerard Pons-Moll, Michael Zollhoefer. and Christian Theobalt. Hdhumans: A hybrid approach for high-fidelity digital humans. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(3):1 2023.
[15] Mare Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer. Gerard Pons-Moll, and Christian Theobalt. Real-time deep dynamic characters. ACM Trans. Graph., 40(4). jul 2021.
[16] Marc Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Moll, and Christian Theobalt. Real-time deep dynamic characters. ACM Transactions on Graphics (TOG), 40(4):1-16, 2021.
[17] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[18] Insung Ihm. Sanghoon Park. and Rae Kyoung Lee. Rendering of spherical light fields. In Proceedings The Fifth Pacific Conference on Computer Graphics and Applications, pages 59 68. IEEE, 1997.
[19] Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image tonge-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1125-1 1 134, 2017.
[20] Zhang Jiakai, Liu Xinhang. Ye Xinyi, Zhao Fuqiang. Zhang Yanshun, Wu Minye, Zhang Yingliang. Xu Lan, and Yu Jingyi. Editable free-viewpoint video using layered neural representation. In ACM SIGGRAPH, 2021.
[21] Wei Jiang. Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from a single video, 2022.
[22] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pages 694-711. Springer, 2016.
[23] Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. Learning-basee view synthesis for light field cameras. ACM Transactions on Graphics (TOG), (566) 2016.
[24] Srinivas Kaza et al. Difserentiable volume rendering using signed distance functions. PhD thesis, Mas- sachusetts Institute of Technology, 2019.
10

---

[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint urXiv:1412.6980 2014.
[26] Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural human performer: Learning generalizable radiance ficlds for human performance rendering. Advances in Neural Information Processing
Systems, 34:24741-24755 2021.
[27] YoungJoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural image-based avatars: Generaliz- able radiance fields for human avatar modeling. In The Eleventh International Conference on Learning Representations, 2023.
[28] Mare Levoy and Pat Hanrahan. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 31-42, 1996.
[29] Ruilong Li. Julian Tanke, Minh Vo. Michael Zollhofer, Jurgen Gall, Angjoo Kanazawa, and Christoph Lassner. Tava: Template-free free animatable volumetric actors. In European Conference on Computer Vision (ECCV). 2022.
[30] Celong Liu, Zhong Li, Junsong Yuan, and Yi Xu. Neulf: Efficient novel view synthesis with neural 4d light field. arXiv preprint arXiv:2 105.071 12, 2021.
[31] Lingjie Liu, Jiatao Gu. Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33, 2020.
[32] Lingjie Liu, Marc Hahermann, Viktor Rudney, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt. Neural actor: Neural free-view synthesis of human actors with pose control. ACM Trans. Graph., 40(6),
dec 2021.
[33] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images, ACM Transactions on Graphics (TOG), 38(4):65, 2019.
[34] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPHI Asia), 1-44::248
Oct. 2015.
[35] Marko Mihajlovic, Aayush Bansal, Michael Zollhoefer, Siyu Tang, and Shunsuke Saito. Keypointnerf: Generalizing image- based volumetric avatars using relative spatial encoding keypoints. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pages 179-197. Springer, 2022.
[36] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng. and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1 14, 2019.
[37] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020, pages 405-421, Cham, 2020. Springer International Publishing.
[38] Parry Moon and Domina Eberle Spenoer. The photic field. Cambridge, 1981.
[39] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. In International Conference on Computer Vision, 2021.
[40] Hao Ouyang. Bo Zhang. Pan Zhang. Hao Yang, Jiaolong Yang, Dong Chen, Qifeng Chen, and Fang Wen. Real-timo neural character rendering with posc-guided multiplane images. In European Conference on Computer Vision, pages 192-209. Springer, 2022.
[41] Sida Peng, Junting Dong. Qianqian Wang. Shangzhan Zhang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Animatable neural radiance fields for human body modeling. ICCV, 2021.
[42] Sida Peng, Chen Geng, Yuanqing Zhang, Yinghao Xu. Qianqian Wang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Implicit neural representations with structured latent codes for human body modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[43] Sida Peng. Yuanqing Zhang, Yinghao Xu, Qianqian Wang. Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. CVPR, 1(1):9054-906) 2021.
[44] Katja Schwarz, Yiyi I Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33:20154-20116, 2020.
[45] Aliaksandra Shysheya, Egor Zakharov, Kara-Ali Aliev, Renat Bashirov, Egor Burkov. Karim Iskakov, Aleksei Ivakhnenko, Yury Malkov, Igor Pasechnik, Dmitry Ulyanov, Alexander Vakhitov, and Victor Lempitsky. Textured neural avatars. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[46] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv:1409 1556, 2014.
[47] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems,
2021.

---

[48]
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias NieBner. Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2437-2446. 2019.
[49] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-awar neural scene representations. In Advances in Neural Information Processing Systems.
pages 1119-1130, 2019.
[50] Shih-Yang Su, Timur Bagautdinov, and Helge Rhodin. Danbo: Disentangled articulated neural body representations via graph neural networks. In European Conference Computer Vision, 2022.
[51] Yang-Yang Su, Frank Yu, Michael Zollhoefer, and Helge Rhodin. A-nerf: Surface-free human 3d pose
refinement via neural rendering. arXiv preprint arXiv:2102.0619 06199. 2021.
[52] Mohammed Suhail. Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light field neural rendering, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognittin, pages 8269-8279, 2022.
[53] Justus Thies, Michael Zollhöfer, and Matthias NieBner. Deferred neural rendering: Image synthesis using neural textures, Acm Transactions on Graphics (TOG), 1-(4))1-12, 2019.
[54] Marco Volino, Dan Casas, John Collomosse, and Adrian Hilton. Optimal representation of multiple view video, In Proceedings of the British Machine Vision Conference. BMVA Press, 2014.
[55] Peng Wang. Yuan Liu, Guying Lin, Jiatao Gu, Lingje Liu, Taku Komura, and Wenping Wang. Progressively-connctee light field network for efficient view synthesis. arXiv preprint arXiv:2207.04465 2022.
[56] Shaofei Wang, Katja Schwarz, Andreas Geiger. and Siyu Tang. Arah: Animatable volume rendering of articulated human sdfs. In European Conference Computer Vision, 2022.
[57] Chung-Yi Yi Weng. Brian Curless, Pratul P. Srinivasan. Jonathan T. Barron, and Ira Saanelmachhe-Shhizrmae Humannerf: Free-viewpoinl rendering of moving people from monocular video. arXiv, 2022.
[58] Daniel Wood, Daniel Azuma, Ken Aldinger. Brian Curless, Tom Duchamp, David H Salesin, and Werner Stuetzle. Surface light fields for 3d photography. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 487-496, 2023.
[59] Minye Wu. Yuehao Wang. Qiang Hu, and Jingyi Yu. Multi-view neural human rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1682-1691, 2020. [60] Donglai Xiang. Timur Bagautdinov, Tuur Stuyck, Fabian Prada, Javier Romero. Weipeng Xu. Shunsuke Saito, Jingfan Guo, Breannan Smith, Takaaki Shiratori, et al. Dressing avatars: Deep photorealistic appcarance for physically simulated clothing. ACM Transactions on Graphics (TOG), (6):11155 2022, [61] Donglai Xiang, Fabian Prada, Zhe Cao, Kaiwen Guo, Chenglei Wu, Jessica Hodgins, and Timur Bagaut- dinov. Drivable avatar elothing: Faithful full-body telepresence with dynamic clothing driven by sparse rgb-d input. In SIGGRAPH Asia 2023, 2023.
[62] Feng Xu, Yebin Liu, Carsten Stoll, James Tompkin, Gaurav Bharaj. Qionghai Dai. Hans-Peter Seidel, Jan Kautz, and Christian Theobalt. Video-based characters: Creating new human performances from multi-view vidco database. In ACM SIGGRAPH 201 Papers, SIGGRAPH '11 pages 32:1-32:10, New York, NY, USA, 2011 1. ACM.
[63]
Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion, 2021.
[64] Jac Shin Yoon, Duygu Ceylan, Tuanfeng Y Wang. Jingwan Lu, Jimei Yang. Zhixin Shu, and Hyun Soo Park. Learning motion-dependent appearance for high-fidelity rendering of dynamic humans from single camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3407-3417, 2022.
[65] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unreasonable elfectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer
vision and puttern recognition, pages 586-595, 2018.
[66] Zerong Zheng, Han Huang, Tao Yu, Hongwen Zhang, Yandong Guo, and Yebin Liu. Structured local radiance fields for human avatar modeling. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 15893-15903, 2022.
12

---

A Appendix Overview
This appendix organized as follows: Sec. [ shows additional results including video results, ablations image-level and patch-level perceptual supervision, ablation study with and without perceptual supervision, ablations on the single surface with the same network capacity, results when using SMPL, comparison with traditional multi-view stereo method; Sec. proovidss information regarding the reproducibility, which includes implementation details, training details, and runtime at inference; Sec. [ discusses the performance changes according to the template mesh quality, and possible alternatives for the template mesh; Sec. presents the societal impacts our work can have; Sec. Io..Flee discusses the limitations of this work.
B
Additional Results
B.1
Video Results
Video results free-viewpoint renderings, novel view and pose synthesis, and comparison with the state-of-the-art baselines the DynaCap dataset [16] can be found at httpps:/vcaai. .mpi- inf mpg mpg.de/proof. httgge//projjcts///oooooo /EEDIFFF We compare our DELIFFAS with the non-real-time hybrid method
Neural Actor 32]. and real-time explicit mesh-based method Deep Dynamic Characters [16].
B.2 Ablation on Image-level and Patch-level Perceptual Supervision
To study the impact of our full image-level supervision, train variant with patch-level perceptual supervision. Here, we use 32 32 patch. In Tab. 3 our approach trained with full-image-leve perceptual supervision (Tab. c) outperforms the patch-level variant (Tab. (3) in terms of the perceptual metrics. In Fig. 7 we ablate the visual impact. Employing full-image-leve supervision leads to better reconstruction of details (e.g., wrinkles). We would like again highlight that image- level supervision with perceptual loss is possible thanks to the fast rendering speed of our method.
leaeellevel
Patch-level Ln Lpere
c. Imago-level
GT
4
L + Eperc (Ours)
Figure 7: Ablation of patch-based and image-level perceptual supervision. Variants with the perceptual supervision (b,c) shows better details than the result with Li-only supervision (a). Our final model which employs the full image-level perceptual supervision (c) outperforms the variant with patch-level supervision (b).
B.3 Ablation Study with and without Perceptual Supervision
excluded the perceptual supervision from the ablation study in the main text (see Tab. 2 Fig. 5 and Fig. 6) to verify the effectiveness of our two-surface design. In Tab. and Fig. g we show the ablation results with perceptual supervision. Similar to the results without perceptual supervision, our full model outperforms other variants.
13

---

Method
PSNR
LPIPS
FID
Image-level with L1 Patch-level with L1 + Lpore
33.27
18.51
23.58
33.24
16.54
18.96
Image-level with L + Lprr (Ours)
33.30
12.85
8.69
Table 3: Ablation image-level and patch-level supervision. Applying the perceptual supervision on the entire image (c) leads to better results than the patch-level perceptual supervision (b) or Li-yyy supervision.
Method
PSNR
LPIPS
FID.
L1
Single-surface
31.34
22.74
28.96
Single-surface + viewing direction
32.19
20.38
23.88
Single-surface two U-Net
31.47
15.43
19.99
d. Two-surface (Ours)
33.27
18.51
23.58
Single-surface with coarse mesh
29.63
27.75
42.84
f. Two-surface with coarse mesh
32.03
22.00
31.43
L1 + Liprrc
g. Single-surface
31.36
16.53
10.90
h. Single-surface viewing direction
31.95
15.96
9.01
i. Single-surface two U-Net
31.26
16.21
9.61
j. Two-surface (Ours)
33.30
12.85
8.69
Single-surface with coarse mesh
29.55
20.05
15.22
1. Two-surface with coarse mesh
32.03
16.13
12.08
Table 4: Ablations on novel view synthesis for the D2 subject with and without perceptual supervision. Our two-surface parameterization performs better than the single-surface variants and the perceptual supervision further boosts the visual quality. In addition, our results with coarse mesh model shows reasonable performance, which confirms the robustness of our method.
B.4 Ablations on the Single Surface with the Same Network Capacity
The number of total parameters is larger in the two-surface model than in the single-surface model as it includes two separate U-Nets for extracting feature map from each surface while the single-surface model only has a single U-Net. The U-Net and MLP architecture used both single and two-surface models are the same. To verify that the performance boost originates from the two-surface design rather than the increase in network capacity, we additionally trained a "single-surfes surface with two concatenated U-Net" variant (see Fig. Tab. 4 c.i). Even though now the network size is the same, our two-surface representation still outperforms the single-surface baseline.
B.5 Using SMPL as the Deformable Mesh Surface
We conduct experiment where the SMPL model as the deformable mesh model (see Fig. [0). One can see that our approach even for this very coarse model still outperforms the baselines proving the flexibility of our representation while better template mesh further improves the results.
B.6 Comparison with Traditional Multi-view Stereo Method
compare with the commercial photogrammetry software, Metashape Fig. II The recon- structed surface is noisy and the texture suffers from ghosting artifacts due to the inaccurate geometry. Also, Metashape takes minutes reconstruct while ours works real-time (31fps) Lastly, our method is animatable, while multi-view stereo methods not.
14

---

a. Single-surface
b. Single-surface viewing direction
c. Two-surface
[ [1] only supervison
GT
d. Single-surface
Single-surface viewing direction
TTwo-surface (Ours)
[C1 + pere supervison
Figure 8: Ablations novel view synthesis for the D2 subject with and without perceptual supervision. The proposed two-surface representation helps recover the real geometry while the single-surface design is limited to the underlying mesh. Perceptual supervision further enhances the quality.
L + Eperc
L
b + Epere (Ours)
Single-surface + two-UNet
Two-surface
GT
Figure 9: Ablations the single surface with the same network capacity. Evaluated the novel view synthesis task for the D2 subject. Even though the network size the "single-surface with two U-Net" variant and single-surface baseline. This again verifies that the performance boost is coming from the two-surface design, and not from using more network.
the same, two-surface representation still outperforms the
15

---

Single-surface
Two-surface
Two-surface
[L1 only
I
[L1 only
SMPL
Original mesh
GT
[C1 percl
[C1 + percl
Figure 10: Ablations on using SMPL as a deformable human model. Evaluated on the novel view synthesis task for the D2 subject. Our method with very coarse SMPL mesh can still recover the real geometry compared to the single surface baseline where the approach can only paint on the given mesh.
C Reproducibility
C.1 Implementation Details
Feature Extractor. The feature extractor for each surface is based on the same U-Net architecture 19]
We use the feature extractor to encode the temporal normal map (T + 1) x 1024 x 1024 3 into the 1024 1024 x 32 feature map. Here, the temporal normal map is the concatenation of the posed normal maps of the motion window [f. - 27. The feature extractor consists of 10 down-sampling layers (with down-sampling rate of two) and 10 up-sampling layers (with up-sampling rate of two). There is a skip connection between every corresponding down and upsampling layer.
Light Field MIP. The light field MLP consists of 8 layer with 256 channel size. ReLU activation is used in between the layers. Input to the light field MLP are the positional encodings of two intersection uv coordinates. Then concatenate the temporal normal map features sampled from the inner and outer surface to the intermediate output of 4th layer. Output of the light field MLP is the 3-channel RGB value.
C.2 Training Details
Deformable Human Model. We leverage the deformable human model of Habermann et al. [6]. ie., DDC. The geometry networks of the DDC model are additionally trained with a Chamfer distance supervision with respect to 4D multi-view stereo reconstructions. Originally, we reported the numbers provided by the authors of DDC in Tab. I We trained TexNet of DDC with exact checkpoint for
16

---

Ours
Metashape
GT
Figure 11: Comparison with MVS method (Metashape). We compare with the commercial pho- togrammetry software, Metashape (.. The reconstructed surface is noisy and the texture exhibits ghosting artifacts. Also, Metashape takes 6 minutes to reconstruct while ours work in real-time (p)) Lastly, our method animatable, while multi-view stereo methods are not.
Method
PSNR1
LPIPS
FID
Method
PSNR
LPIPS
FID.
DDC (paper)
32,96
20.07
27.73
DDC DDC (paper)
28.03
30.43
38.37
geometry network)
32,66
DDC geometry aetwork)
43.43
61.92
Ours
33.30
12.85
8.69
Ours
27.95
26.59
26.16
(a) View synthesis error on 1)2
(b) Motion synthesis error on D)2
Table We trained TexNet of DDC with our exact checkpoint for the geometry networks (EGNet
and DeltaNet) that are trained with Chamfer distance to alleviate any other influence. We found that our geometry checkpoint has even lower performance than the original one. Note that our representation clearly outperforms DDC.
the geometry networks trained with Chamfer distance supervision (EGNet and DeltaNet) alleviate any other influence. We found that geometry checkpoint has lower performance than the original one. Note that our representation clearly outperforms DDC (see Tab. 5). Morcover, our ablation "single surface + viewing direction' (Fig Tab. 2 can be considered as variant of DDC further confirming the superiority of our appearance representation.
DELIFFAS. Both the feature extractors and light field MLP are trained in an end-to-end manner (25]. We used the learning rate of 2 x 10-4, We train on a single RTX 8000 48G GPU with a single batch size. We supervise our approach by minimizing the following loss A Li + - where Li and Lpere are the L, loss and perceptual [22) loss, respectively. Åι and Apere are their weights. For the first 940k iterations, X is set to one, and Xpere is set to zero. Then, we set A₁ and Apere so that the magnitude of each loss term is roughly the same and train for additional 350k iterations.
using Adam optimizer
C.3 Runtime Inference
We report our runtime on the novel pose synthesis task D2 subject. DELIFFAS runs at 31 fps for rendering single 1K (940 x 1285) image single A100 GPU with an Intel Xeon CPU, The detailed breakdown of runtime is as follows.
Our pipeline can be divided into three stages: (1) obtaining UV (2) feature map generation (CNN)
(3) MLP forwarding to get the final color value. (1) Obtaining UV: We utilize GPU accelerated rasterizer to render the image space UV map, which takes 3.58 ms. (2) Feature map generation
(CNN): Our U-Nets generate normal feature maps in 18.85 ms. (3) MLP forwarding to compute the final color: We take 6.78ms to bi-linearly sample the feature corresponding to each pixel, and then the light field MI.P takes 2.52ms to compute the color value. In total, DELIFFAS takes 31.73
ms to render a single 1K image.
17

---

Single Single-surfaec surfacc
Twoossufacc
Two-surface
[L1 only
[L - only
Coarse mesh
Original mesh
GT
[ + Sperel
[C, + peerel
Figure 12: Ablations the deformable template quality with and without perceptual supervision. Evaluated on the novel view synthesis task for the D2 subject. Even when using the coarse mesh, our two-surface design enables to generate visual quality similar to the rendering result using original mesh. Employing the perceptual supervision further improves the quality.
Note that we assume the output of the deformable character model, i.e., DDC [16], is already available training and inference time. DDC is also a real-time method so coupling DDC and our method would not deteriorate the performance much.
We would like to highlight that our method achieves the best of two worlds: in real-time the explicit mesh-based method DDC (25 (25/ps), ps), while at the same time achieving superior and comparable visual quality the hybrid method NA (0.2fps) and HDHumans (0.3fps), respectively.
When tested on an A40 graphics card, we achieve real-time speed 26 fps. Furthermore, most of our computational budget is taken by the U-Net feature map extraction 60%) and deterministic bi-linear sampling features (0%)). use a standard U-Net architecture and Tensorflow's 's bi-linear sampling, but more efficient architecture and sampling implementation be explored further reduce the runtime.
D Discussion on the Template Mesh Quality
Rendering Quality, Our performance degrades a coarser mesh used. However, note that our method with coarse geometry even SMPL mesh can still recover the real geometry compared the single surface baseline where the approach can only paint on the given mesh (see Fig. 6 Fig. 10 and Fig. 12). Also, it would be an interesting research direction if we could jointly refine the underlying mesh during training and improve the rendering quality.
Possible Alternatives to the Deformable Human Model. The deformable human model we used in
main text is DDC [6]. Alternatively, one can use the SMPL model (see Fig. 10) or SMPL+D to
deal with the loose clothing. Also, we can use recent avatar NeRF works [64 21
42 to compute the
canonical mesh (template mesh) and then use skinning- deformation to obtain the deformed
mesh for the new pose.
18

---

E
Societal Impacts
Our research presents potential societal impacts that should be considered. Our method can create avatars using only RGB supervision, which has the potential to democratize immersive VR expe- riences. Unlike traditional methods that rely on costly setups and specialized professionals, our approach offers accessibility to wider range of individuals. Additionally, digital doubles generated through our techniques can used to perform dangerous stunts, reducing risks for human actors. Moreover, individuals can engage in virtual exploration using their personalized avatars.
However, it is important acknowledge the potential negative consequences associated with method. The generation of fake media poses significant risk, potentially leading to public misin- formation and eroding trust in media content. We recognize these concerns and emphasize the need for public awareness, responsible use, and disclosure. We strive and strongly hope that our work is applied in ways that positively impact society.
F Limitations
Although our method achieves state-of-the-art results in terms of visual quality and runtime, it is not free from limitations. For now, we consider the deformable surface representation given as an input and our method builds up on it. However, surface deformation and image synthesis can be treated jointly and help to improve each other. Inspired by that, future work could extend our deformable two-surface light field representation such that it can backpropagate into the geometry directly and, thus, potentially refine it throughout the training process. Moreover, the face and hand not animatable. Extending versatility of our method and enable many interesting applications such as gaming, and VR. Very complex clothing (e.g., highly glossy garment) might be challenging. though, in contrast many related works, we demonstrate that loose clothing can work while most other methods only operate under the assumption tight clothing. The input our system is temporal normal map and it sufficient enough to fully describe the subject's clothing state which can result in rather blurry images. This is why we employed perceptual supervision so that we can recover the fine details. However, would be a promising direction to adopt the generative models (e.g., GAN, VAE) to generate even more photorealistic details.
work to support the face and hand animation would enhance the
19


Reviews:
[{"review_id": "Tggkis6aKM", "reviewer": "Reviewer_QJHp", "summary": "The paper presents a real-time approach for generating controllable and photorealistic digital human avatars. The main contribution of the paper is to learn the texture of the avatar using a light-field representation. In contrast to other continuous representations (e.g., NeRF), the proposed method can predict the color of a ray using a single MLP call. This allows faster rendering of the images which in turn allows using perceptual losses during training and yields significantly better synthesis results. The paper mainly extends the Deep Dynamic Characters (DDC) method and replaces its TextureNet with the proposed light-field representation. The experiments are performed on the DynaCap dataset where the proposed method is shown to outperform other real-time methods while being on par with the non-realtime methods.  ", "questions": "### Checkerboard artifacts:\n- What are those checkerboard artifacts in Fig. 4? With such artifacts it is very easy to identify that the images are computer generated, hence it raises a question regarding paper's claim of `photorealistic` synthesis. Perhaps tone down the photorealistc claim if these artifacts are not preventable.  \n\n### Experiments\n- The best setting of the proposed method is with $L_{perc}$, I wonder why the ablation studies are performed without this loss? What happens if we use $L_{perc}$ with `Single surface + viewing direction` setting? \n\n### Overview figure\nWhy the red dot is at a different location in the feature maps? Shouldn't it be consistent with the temporal normal maps? \n", "limitations": "The limitations of the paper are not highlighted in the paper. The authors should add the during the rebuttal.  \n\nPotential limitations are:\n1. Requirement of an initial 3D scan of the person as required by DDC, which is not easy to obtain without having access to sophisticated hardware. \n2. Long training sequences. \n3. Use in deep fakes. \n4. Privacy concerns, etc. \n", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The use of the light slab method for learning the texture details of the avatar is novel and makes sense. I believe this has the potential to be used in other methods as well where faster rendering speed is required. \n- Since the color for each ray can be predicted in one shot, the proposed method allows rendering full images during training. Hence, perceptual losses can be used and are shown to help significantly (LPIPS: 18.51 vs 12.85) \n- The proposed method achieves results on par with the state-of-the-art while being 100x better. \n- The paper is well-written and easy to read, though some typos need to be fixed  ", "weaknesses": "### Novelty \n- My main concern with the paper is its limited novelty. The paper builds on DDC and mostly replaces its TextureNet with the proposed light-field-based MLPs. It is unclear whether better results are due to better engineering or the proposed light-field-based approach. Is the implementation of DDC and the proposed method exactly the same with texture nets being the only difference between the two? \n\n### Missing Experiments\n- I would have liked to see an experiment in which light-field is replaced with vanilla NeRF while having everything else exactly the same.  \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback to further improve our work. Please see our visual and quantitative ablations in the global response-rebutal.pdf.\n* * * \n**Novelty**\n\nWe would like to emphasize that this work is about finding an appearance representation that is, both, efficient and supports high quality.  To this end, we present the deformable two-surface light field parameterization. We assume a deformable mesh model is given, this can be either DDC or as our ablation shows also a coarser deformable mesh or even SMPL mesh would work reasonably well (see rebuttal.pdf-Fig.2,4). Originally, we reported the numbers provided by the authors of DDC. For the rebuttal, we trained DDC’s TexNet with our exact checkpoint for the geometry networks (EGNet and DeltaNet) to alleviate any other influence. We found that our geometry checkpoint has even a lower performance than the original one. Note that our representation clearly outperforms DDC’s results (see below). Moreover, our ablation ‘single surface + viewing direction’ (main paper Fig.5-b,Tab.2-b) can be considered as a variant of DDC further confirming the superiority of our appearance representation. \n\n| Novel view                  | PSNR      | LPIPS     | FID      |\n|-----------------------------|-----------|-----------|----------|\n| DDC (paper)                 | 32.96     | 20.07     | 27.73    |\n| DDC (same geometry network) | 29.50     | 32.66     | 44.18    |\n| **Ours**                    | **33.30** | **12.85** | **8.69** |\n\n| Novel pose                  | PSNR      | LPIPS     | FID      |\n|-----------------------------|-----------|-----------|----------|\n| DDC (paper)                 | **28.05**     | 30.43     | 38.37    |\n| DDC (same geometry network) | 26.59     | 43.43     | 61.92    |\n| **Ours**                    | 27.95 | **26.59** | **26.16** |\n\n\n\n* * * \n**NeRF ablation**\n\nThis might be challenging as rendering the 1K image with volumetric rendering takes very long, e.g., Neural Actor and HDHuman take 4~5 seconds, which makes the perceptual supervision on the entire image inefficient.\n\n* * * \n**Checkerboard artifacts**\n\nThis is not a checkerboard artifact. The D2 subject has a stripe pattern on the T-shirt if closed up (see GT of supplementary document-Fig.1 and rebuttal.pdf-Fig.3) and our approach learned to reproduce it. This actually shows our ability to capture very fine details.\n\n* * * \n**Why ablation study without L_perc? / Additional Ablation “single surface + viewing direction + perceptual loss”**\n\nWe conducted this ablation to demonstrate that the proposed two-surface representation in isolation with the perceptual supervision achieves improved visual accuracy, which is confirmed by our experiment. Adding the perceptual supervision further improves visual quality.\nFor the rebuttal, we further added ablations with perceptual supervision as requested by the reviewer (see rebuttal.pdf-Fig.1,2,Tab.1). The variant with “single surface + viewing direction + perceptual loss” is shown in rebuttal.pdf-Fig.1-e and Tab.1-h. \nIn summary, the combination of our proposed two-surface representation and the perceptual supervision achieves the best result. \n\n* * * \n**Limitations**\n\nWe will include the suggested limitations in the revision. Regarding the first limitation (requirement of an initial 3D scan which is not easy to obtain), we would like to note that this assumption is not too restrictive as competing methods (we compare to) rely on a multi-view sequence. A template from multi-view imagery could be acquired by methods like multi-view PIFu.\nRegarding the second limitation (long training sequences), this is rather a pro than a con. Most methods are not even able to train on such long sequences since their representation fails to optimize parameters that can cover such diverse poses and as a consequence produce very blurry results (see main paper-Fig.4 NB, A-NeRF results).\nConcerning deep fake generation and privacy concerns, we will add such a discussion. \nMoreover, we would like to highlight that we discuss limitations in the supplemental document. However, for the final version we will incorporate this paragraph into the main manuscript.\n\n***\n**Overview figure**\n\nThe location of the red dots should be consistent with the temporal normal maps. We will modify this in the revision."}, {"review_id": "18JJzHHQGz", "reviewer": "Reviewer_dLaT", "summary": "This paper aims to learn digital human body avatars from multi-view videos. To achieve both photorealism and fast inference speed, the authors introduce a new representation based on a surface light field. The surface light field is attached to a drivable human mesh model, and is conditioned on skeleton motion. Such a representation can be easily defined on texture atlas, and efficiently rendered with standard rasterization pipelines. Overall, the proposed representation allows perceptual supervision and real-time rendering, resulting in superior performance over existing baselines. ", "questions": "None", "limitations": "The authors discuss the limitations and potential societal impact in the supplemental document. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method enables real-time rendering of full-body human avatars with a resolution of 1K. This is a nice property for many down-stream interactive applications. \n\n2. The proposed representation further allows perceptual supervision on the full image, which is crucial for learning high-quality appearance details like cloth wrinkles. \n\n3. All important implementation details are adequately discussed in the main paper and the supplemental document. For example, in line 179-180, the authors describe how they handle corner cases where a ray only intersects with the outer surface.", "weaknesses": "1. Although the proposed representation is based on a light-field defined by two deformable surfaces and demonstrates superior performance over the single-surface baseline Sec.4.3, it is still tightly bounded to the underlying mesh model. According to Line 149, the shell bounded by two surfaces has a thickness of only 3 cm, which is relatively small. This means that the underlying mesh model should be a close approximation of the real avatar surface. Therefore, I am suspicuous whether other human body templates (like SMPL) can be used in the proposed method, although the authors claim in Line 122 that \"any other deformable surface representation could be used as well\". \n\n\n2. Although the proposed two-surface representation is nice for modeling dynamic texture, the geometric deformation is not modeled. In Line 118, the authors mention that they assume \"a skeletal motion-depdent deformable mesh surface is given\". This can be obtained for the task of novel view synthesis, as mesh tracking is a key step in data preprocessing. However, this is very challenging for novel pose synthesis, especially for loose garments like a long dress. Unfortunately, I can't find any details about how they obtain the mesh model for novel poses in the paper. \n\n\n3. The authors claimed in Abstract that the combination of photorealism and inference speed \"still remains unsolved\", which I think is not true. Some recent works have already achieved real-time neural character rendering with high photo-realisitc quality. For example, Ouyang et al. [a] proposed to use multi-plane image (MPI), which is also a light field representation, to render an animatable character in real-time. Ouyang et al. also applied perceptual supervision on full images to learn high-quality appearance details. Therefore, I think this submission should cite, discuss, and perhaps compare with this highly relevant work [a]. \n\n[a] Ouyang et al. Real-Time Neural Character Rendering with Pose-Guided Multiplane Images. ECCV 2022. \n\n\n4. In Line 199-200, the authors mention that the blurry results trained with L1 loss is due to the one-to-many mapping problem. However, their solution, i.e., applying perceptual supervision, does not resolve this problem; in other words, the one-to-many problem still exists after applying the perceptual loss. I guess this is the main reason for the jittering texture in the supplemental video (00:14-00:29). \n\n5. The novel pose sequences for the long skirt case (supplemental video 00:56-00:59) are too short.\n\n\nTypos: Line 278: In Fig.5, **We** --> In Fig.5, **we**\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback to further improve our work. Please see our visual and quantitative ablations in the global response-rebutal.pdf.\n* * * \n**Bounded to mesh**\n\nWe conducted an experiment where we use the SMPL model as the deformable mesh model (see rebuttal.pdf-Fig.4). One can see that our approach even for this very coarse model still outperforms the baselines proving the flexibility of our representation while a better template mesh further improves the results. We will include this experiment in the camera ready version. \n* * * \n\n**Details about the mesh model**\n\nWe used the off-the-shelf DDC [1] as our deformable surface model as mentioned in Section 3.1. This model takes a (novel) skeletal motion as input and generates the posed and non-rigidly deformed template while being supervised solely on multi-view imagery. We will add more details about DDC in the final version. \n\n* * * \n**Claim with respect to Ouyang et al.** \n\nAlthough Quyang et al. achieve impressive and photorealistic results, they only allow minimal pose changes and mostly overfit to the specific recording as also stated in their limitations. In contrast, our method can handle arbitrary poses at test time and, thus, is a controllable character representation. Moreover, they mostly show frontward facing camera movements, while our method supports full 360 camera rotations around the human subject. We will adjust the claims while considering Quyang et al. and discuss the differences between Quyang et al. and ours in the revision.\n\n* * * \n**Solving the one-to-many mapping**\n\nLike most works we do not explicitly model these stochastic effects in clothing, but we found that the perceptual loss effectively addresses them (see main paper-Fig.5 c,d) while some small residual might still remain. Also, since we can render the entire image thanks to the fast rendering speed,  generative models (e.g., GAN, VAE) could be incorporated to model this stochastic process. We leave this as future work.\n\n* * * \n**Sequence too short**\n\nWe will increase the length of the sequence.\n\n[1] Habermann et al. “Real-time deep dynamic characters.” In TOG 2021."}, {"review_id": "1j1oqEwX6N", "reviewer": "Reviewer_cQc3", "summary": "The paper introduces DELIFFAS, an innovative method for generating controllable and photorealistic digital human avatars in real-time. This system utilizes a deformable two-surface representation to parameterize a surface light field, deforming two surfaces according to a deformable mesh model, allowing the light field to be driven by skeletal motion. This approach significantly increases inference speed, enabling the entire image to be rendered during training, unlike previous methods. This faster rendering time also allows for full-image level perceptual supervision, in contrast to earlier approaches, which could only supervise individual pixels or small patches due to slow runtime.", "questions": "- How does the DELIFFAS model perform when exposed to extreme poses or movements not well-represented in the training data? Are there specific types of movement or pose it struggles to accurately represent?\n\n- Could you elaborate on the computational resources required for real-time implementation of the DELIFFAS model? A100 is a quite heavy GPU configuration, what happens if tested on other GPU configurations for the performance?\n\n-  How would the system handle more complex or unusual clothing types, especially those not well-represented in the training data?\n\n- The method seems to rely heavily on the quality of the underlying mesh model.", "limitations": "Discussed above, in the weakness section. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Key strengths of the paper:\n\n- The proposed method DELIFFAS can generate controllable and photorealistic human avatars in real-time.\n\n- The use of a deformable two-surface representation to parameterize a surface light field improves both accuracy and efficiency of appearance synthesis.\n\n- The efficient design of the neural architecture and light field representation allows for integration into the graphics pipeline, resulting in real-time performance.\n\n- Due to the high inference speed, the system can render the entire image during training and employ full-image level perceptual supervision.", "weaknesses": "Some potential weaknesses of the paper:\n\n- Limitations in Pose Variety: It is unclear how well the model would perform with extremely unusual or rare poses, or if it would be able to accurately generate and maintain details under these conditions.\n\n- Dependency on Underlying Mesh Quality: While the authors have demonstrated that their method can tolerate coarser geometry, the accuracy and realism of the output may still depend on the quality of the underlying mesh.\n\n- Robustness to Complex Clothing:  While the authors mention that their method can handle \"challenging loose garment types,\" there may be limitations when dealing with more complex or unusual clothing, especially if such examples are not well represented in the training data. This can be often the case where common clothing could exhibit highly glossy or specular lighting effects such as silk. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback to further improve our work. Please see our visual and quantitative ablations in the global response-rebutal.pdf.\n* * * \n**Pose variety**\n\nTypically our method is robust to rather challenging poses, however, completely out-of-distribution poses such as handstand might not work since cloth deformation can be very large, i.e., shirt pulls down. We encourage the reviewer to look at the publicly available test set of the DynaCap sequences, which comprise a large variety of poses.\n\n* * * \n**Dependency on mesh quality**\n\nOur performance can degrade if a coarser mesh is used. However, note that our method with coarse geometry or even SMPL mesh can still recover the real geometry compared to the single surface baseline where the approach can only paint on the given mesh (see main paper-Fig.6, rebuttal.pdf-Fig.2,4).\nAlso, it would be an interesting research direction if we can jointly refine the underlying mesh during training and improve the rendering quality. However, it is important to note that the deformable character model is also solely learned from multi-view video, thus, input assumptions are the same compared to competing methods.\n\n* * * \n**Robustness to complex clothing**\n\nVery complex clothing (e.g., highly glossy garment) might be challenging, though, in contrast to many related works, we demonstrate that loose clothing can work while most other methods only operate under the assumption of tight clothing. We will discuss this in the limitations.\n\n* * * \n**Computational resources**\n\nAs we stated in the main paper L41-45, our method runs at 31 fps for rendering a single 1K image (940X1285) on a single A100 GPU with Intel Xeon CPU. \nWhen tested on A40, we achieve the real-time speed of 26 fps. Also, please note that our method runs X100 faster than the SOTA methods NA and HDHuman when the same computing resource is given.\n\nFurthermore, most of our computational time is coming from the U-Net feature map extraction (\\~60%) and deterministic bi-linear sampling of features (\\~20%). We use a standard U-Net architecture and Tensorflow’s bi-linear sampling, but more efficient architecture and sampling implementation can be explored to further reduce the runtime.\n\n* * * \n**Clothing not in training data**\n\nOur method is a person/clothing specific method and we do not claim generalization to new clothing (as most previous work does).\n"}, {"review_id": "6XXt5F1FDW", "reviewer": "Reviewer_LHHA", "summary": "This work proposes a method for human avatar reconstruction from multiview video data. This work leverages deformable light fields to model the geometry and texture. Experiments show that this method outperforms the existing methods in terms of novel view and novel pose synthesis.", "questions": "The training data are multiview videos, is the proposed method possible to be trained on monocular videos?", "limitations": "Addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using light field to improve computation efficiency is novel and well-motivated. It can overcome some of the drawbacks of NeRF, such as the heavy computation cost for point sampling.\n2. Using mesh as geometry representation is also reasonable as it is naturally compatible with conventional rendering engines, which could open up many application possibilities.", "weaknesses": "1. Although the framework is novel, the idea of using differentiable rasterization for animatable human avatar creation has already been explored in prior work[1], please explain the difference between this work and [1].\n2. There may exist a risk of overfitting because this work only experiments on one dataset, it could be better to add more experiments on the commonly used benchmark dataset such as ZJU-Mocap.\n3. Efficiency is one of the most important contributions of this work, the authors could report more numerical results to verify its rendering speed, and it’s better to compare it with [1].\n4. The setting for ablation study (line 269) is confusing, I don’t understand why other variants are not trained with perceptual loss. I think this loss term should not be skipped except for studying the necessity of optimization objectives.\n\n[1] UV Volumes for Real-time Rendering of Editable Free-view Human Performance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback to further improve our work. Please see our visual and quantitative ablations in the global response-rebutal.pdf.\n\n* * * \n**Difference to UV volumes** \n\nPlease note that Chen et al. [1] is a concurrent work presented in CVPR 2023, which was held after the NeurIPS submission. The difference between [1] and ours is as follows: [1] is a single surface-based method that is tightly bounded to the underlying mesh (optimized uv map). However, ours can go beyond the underlying mesh and recover the real geometry that lies between the two surfaces (see main paper-Fig.5,6, rebuttal.pdf-Fig.2,4). Also, [1] supervises the uv map with the DensePose prediction. Therefore, as mentioned in their limitation section, they can only handle clothing types that roughly fit the human body. On the other hand, ours can handle loose garments such as skirts. We will cite this work and discuss the differences in the revised version.\n\n* * * \n**Risk of overfitting**\n\nDynaCap is also a well-established benchmark used by other works [2,3,4,5] and known to be significantly more challenging than ZJU-Mocap in terms of pose variety. For example, the test sequence comprises around 7000 frames with challenging poses. Further, our results show that methods that perform reasonably well on ZJU-Mocap obtain significantly more blurred results on DynaCap due to the increased pose variety (see main paper-Fig.4 NB, A-NeRF results).\n\n* * * \n**Numerical results for runtime performance**\n\nWe report the performance comparison with Chen et al. on a single A100 GPU. \nDue to the limited time, we report the performance of Chen et al. according to their paper-Tab.4.\nChen et al. achieves 18 fps on the novel pose synthesis task while ours achieves 31 fps. Although there are some differences in the exact pipeline, the pipeline of Chen et al. and ours can be divided into three stages: (1) obtaining UV (2) feature map generation (CNN) (3) MLP forwarding to get the final color value. \n\n**(1) Obtaining UV**\n\nChen et al. generates UV feature volume using sparse CNN (48.78ms). Density along the camera ray is computed (7.08ms). Then the image space uv feature map is volume-rendered (1.73ms). The UV mlp converts the feature into uv coordinates (1.53ms). This stage sums up to 59.12 ms.\n\nWe utilize GPU accelerated rasterizer to render the image space uv map, which takes 3.58 ms.\n\n**(2) Feature map generation (CNN)**\n\nChen et al. takes 7.52ms to compute the neural texture map. \n\nOur U-Nets generate normal feature maps in 18.85 ms.\n\n**(3) MLP forwarding to compute the final color**\n\nChen et al. takes 1.60 ms to compute color using MLP.\n\nWe take 6.78ms to bi-linearly sample the feature corresponding to each pixel. And then the light field MLP takes 2.52ms to compute the color value.\n\n\nIn total, Chen et al. takes 68.23 ms and ours take 31.73 ms. We will include this comparison in the revision.\n\n* * * \n**Ablation study**\n\nThe reason why we included the ablation results generated only with L1 supervision is because otherwise it would be unclear whether the proposed two-surface representation or the perceptual supervision improves the result quality.\nWe also included the ablation results with perceptual supervision as well in the rebuttal.pdf-Fig.1,2,Tab.1. Similar to the results without perceptual supervision, our full model outperforms other variants.\n\n* * * \n**Monocular videos**\n\nAlthough we could not try due to the limited time, we expect that the performance would naturally degrade as the system has less chance to observe the multi-view information. We will leave this as future work.\n\n[1] Chen et al. “UV Volumes for Real-time Rendering of Editable Free-view Human Performance.” In CVPR 2023.\n\n[2] Habermann et al. “Real-time deep dynamic characters.” In TOG 2021.\n\n[3] Liu et al. “Neural actor: Neural free-view synthesis of human actors with pose control.” In TOG 2021.\n\n[4] Zheng et al. “Structured Local Radiance Fields for Human Avatar Modeling.” In CVPR 2022.\n\n[5] Peng et al. “Implicit Neural Representations with Structured Latent Codes for Human Body Modeling.” In TPAMI 2023.\n"}, {"review_id": "YKJBR0yaDg", "reviewer": "Reviewer_qBLT", "summary": "The paper introduces a method for real-time rendering of animatable full-body avatars. The main contribution is a hybrid mesh-lightfield representation: two surfaces are produced with an off-the-shelf motion-to-surface model, and normals of those surfaces are then used as input to two UNet, the output of those is fit into the lightfield MLP as features. Method is near real-time and runs at 30FPS on a GPU for 1K resolution. Quantitative and qualitative results over recent baselines demonstrate that the method is on par with state-of-the-art.", "questions": "- Does the mesh model take as input anything apart from the pose? Would it be sufficient to capture non-pose-dependent deformations? \n- Do you expect the quality of the method to degrade on grazing angles? \n- Is the number of parameters between a single- and two- layered model the same? In the two-layer case, there are two UNets, which could also explain the boost in the performance?\n- Could you please confirm that the qualitative comparison (Figure 6) is done consistently for both single- and two-layer models trained with/without perceptual loss? Might be a good idea to be more explicit about this in Table 1 as well. \n", "limitations": "Authors did not really discuss limitations. \nSuggestions:\n- inability to capture realistic dynamics due to information-defficient inputs.\n- artifacts on grazing angles due to incoherent features between the two layers.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Paper is well-written and is easy to follow. \n- Proposed method seems technically sound, and the two-layer surface formulation fits really well with the lightfield-based neural rendering. \n- Apart from the quality, proposed formulation runs at 31fps for a 1K resolution image, and only requires 1 MLP evaluation per ray. \n- Quantitative and qualitative evaluation is thorough, the choice of baselines is reasonable (although there are some potential issues, please see below). Ablation study is present and seems to suggest that the proposed two-layer formulation is less sensitive to the quality of the mesh (which can be considered a contribution in itself).\n", "weaknesses": "- There are not too many details on how the deformed mesh is obtained, but assuming it is only conditioned on the motion, it is unclear whether providing normals of the resulting mesh should be sufficient to reconstruct the full image: it would seem that the state of person's clothing cannot be fully described by the pose or sequence of poses. \n- On a similar note, it looks like the underlying mesh actually seems to have some measurable effect for the method performance: judging from numerical results in Table 2)? From Figure 6, it is unclear if the quality comparison is done for both models trained with perceptual loss or not? Yet, authors do not really provide any details on how to build the underlying motion-to-mesh model. \n- Claim \"This allows perceptual supervision ... compared to previous approaches\" is incorrect, e.g. [Habermann'15], [Bagautdinov'21], DVA [Remelli'22] should support full-image supervision with arbitrary perceptual losses. Generally, it is a bit unclear how much of the qualitative improvements are coming from the use of perceptual loss (which I would not consider a contribution of this work). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback to further improve our work. Please see our visual and quantitative ablations in the global response-rebutal.pdf.\n\n* * * \n**Details about the mesh**\n\nThe method we use to compute the deformed mesh is DDC [1] and it is only conditioned on the motion history. We will include more details about it in the supplementary material. It is true that the normal map is not sufficient enough to fully describe the subject’s clothing state, which can result in rather blurry images. This is why we employed perceptual supervision so that we can recover the fine details.\n\n* * * \n**Quality of mesh**\n\nOur performance degrades if a coarser mesh is used. However, note that our method with coarse geometry or even SMPL mesh can still recover the real geometry compared to the single surface baseline where the system can only paint on the given mesh (see main paper-Fig.6 and rebuttal.pdf-Fig.2,4). Also, it would be an interesting research direction if we can jointly refine the underlying mesh during training and improve the rendering quality.\nThe mesh model we used is DDC. Alternatively, one can use the SMPL model (see rebuttal.pdf-Fig.4) or SMPL+D to deal with the loose clothing. Also, we can use recent Avatar NeRF works to compute the canonical mesh (template mesh) and then use skinning-based deformation to obtain the deformed mesh for the new pose. We will include more details about DDC and other alternatives in the supplementary material. \n\nFor Fig.6, both models are trained without a perceptual loss. We included the result only with L1 supervision for the ablations because otherwise it would be unclear whether the proposed two-surface representation or the perceptual supervision improves the result quality. We will make this more clear in the revision. We also included ablations with perceptual supervision in the rebuttal document (see rebuttal.pdf-Fig.1,Fig.2,Tab.1)\n\n* * * \n**Claim about perceptual supervision**\n\nWe did not claim that we are the only work that employs the perceptual supervision on the entire image. We claimed that we can employ it thanks to our fast rendering speed while coordinate-based MLP methods (e.g., NeRF methods) cannot or can only employ perceptual supervision on the small patch due to its slow rendering speed. As shown in the supplementary material-Fig.1 and rebuttal.pdf-Fig.1, the perceptual supervision on the entire image allows recovering the fine details.\n\n* * * \n**Input to the mesh model**\n\nThe only input to the deformation surface model is the pose history. This is not sufficient to capture fine details, and this is why we employ perceptual supervision to represent such details.\n\n* * * \n**Grazing angles**\n\nWe have already shown the rendering results on the grazing angles in the supplementary material video (00:14-00:30, 00:45-00:54, 02:17-02:27), where the camera is freely rotating around the subject. However, we have found no notable artifact or quality degradation on grazing angles.\n\n* * * \n**Number of parameters**\n\nThe number of total parameters is larger in the two-surface model as it includes two separate U-Nets for extracting feature map from each surface while the single-surface model only has single U-Net. The U-Net and MLP architecture used in both single and two-surface model are the same.\nTo verify that the performance boost is coming from the two-surface design, we additionally trained a “single-surface with two U-Net” variant (see rebuttal.pdf-Fig.5,Tab.1-c,i). Even though now the network size is the same, our two-surface representation still outperforms the single-surface baseline.\n\n* * * \n**Consistent experiments in Figure 6 and Table 1**\n\nThe number reported in Table 1 (comparison with other works) is computed with our final model with full supervision (two-surface + L1 + perceptual). The variants in Table 2 (ablation study) are all trained with L1-only supervision except our final model (Tab.2-d). The qualitative results on smooth mesh in Fig.6 is also trained with L1-only supervision for both the single and two-surface models. Again, excluding the perceptual supervision from the ablation study is to verify the effectiveness of our two-surface design. Lastly, we additionally verify the effect of perceptual supervision on the entire image in the supplementary document Fig.1. We will make it more clear in the text and figure caption that the results in Fig.6 are generated with models trained only with L1 supervision. \n\nAlso, we included the ablation results with perceptual supervision as well in the rebuttal.pdf-Fig.1,2,Tab.1. Similar to the results without perceptual supervision, our full model outperforms other variants.\n\n* * * \n**Limitations**\n\nThank you for the suggestion. Regarding the limitation 1, it is true that the normal map is not sufficient enough to fully describe the subject’s clothing state which can result in rather blurry images. This is why we employed perceptual supervision so that we can recover the fine details. However, it would be a promising direction to adopt the generative models (GAN, VAE, …) to generate even more photorealistic details. Regarding the limitation 2, although we have found no notable artifacts on the grazing angles (as can be seen in the free-viewpoint rendering result in the supplementary video), it would be an interesting research direction to further refine the designs for the corner cases. We will include these in the limitations section of the revision.\n\n[1] Habermann et al. “Real-time deep dynamic characters.” In TOG 2021.\n"}]

Author rebuttal:
We thank all the reviewers for their time and effort to review our work and to further improve it. We address individual concerns in the reviewer-specific rebuttal dialogue and provide a pdf (rebuttal.pdf) comprising additional visualizations and experiments.





Meta-review: The manuscript received uniformly positive evaluations from all reviewers. Although initial concerns were raised regarding incremental adjustments to the DDC framework, the influence of underlying geometry, and the experimental demonstrations, these issues were satisfactorily addressed during the rebuttal phase. Consequently, the AC recommends the acceptance of the manuscript.
Decision: Accept (poster)