SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Metrics. We evaluate our performance using the peak signal-to-noise ratio (PSNR). However, human perception cannot be fully reflected with this metric since a very blurry and unrealistic result can still lead to a low error [65]. Therefore, we additionally employ the learned perceptual image patch similarity (LPIPS) [65], and the Fréchet inception distance (FID) [17], which are similar to the human perception. We generate and evaluate results at  $1K$  resolution  $(1285 \times 940)$ . Every metric is computed by averaging across the entire sequence using every  $10th$  frame. Four held-out cameras (7, 18, 27, 40), which are uniformly sampled in the space, are used for the testing.
4.1 Qualitative Results
We first present the qualitative results on the novel view synthesis task in Fig. 3-(A) and on the novel pose synthesis task in Fig. 3-(B). For both tasks, we show the synthesis result of the subject in two different poses viewed from two different viewpoints. Our method synthesizes images with fine-scale details including the facial features and wrinkles of the clothing for both tasks. Note that our method can even generate plausible results of the subject in a challenging garment type, i.e., loose skirt. Furthermore, we can synthesize high-quality renderings of the subject under difficult testing poses. These results confirm the versatility as well as the generalization ability of our method.
We compare our method with the state-of-the-art methods concerning novel view and pose synthesis. We choose baselines from four different classes, which are volume rendering, explicit mesh, hybrid, and point cloud-based methods. For the volume rendering-based method, we compare with Neural Volumes (NV) [33], which learns a volumetric representation. Deep Dynamic Character (DDC) [16] is an explicit mesh-based method, where the network learns to render the neural texture obtained by rasterizing the explicit mesh. Neural Actor (NA) [32], Neural Body (NB) [43], and A-NeRF [51] are hybrid methods that combine the human prior, i.e., naked human body model [34], and skeleton,
Figure 3: Qualitative results for novel view and poses. Our method achieves photorealistic rendering quality while running in real time. We are able to recover the fine-grained details including the facial features and clothing wrinkles. The high-quality synthesis results on the very challenging loose garment, i.e., skirt, and the complicated testing poses show the versatility of our method.
Figure 4: Comparisons on the novel view and pose synthesis task on the  $D_{2}$  subject. Our method synthesizes realistic and delicate details in real time (31 fps). Note that we achieve similar or better quality as offline approaches and show drastically improved quality compared to real-time methods.
Figure 5: Ablation of each design choice on the novel view synthesis task. All the variants except (d) are trained only with  $L_{1}$  loss. Although conditioning on the viewing direction (b) improves the quality, single-surface models (a,b) are tightly bounded to the underlying mesh. On the other hand, our proposed method (c,d) can go beyond the mesh and recover the real geometry. Additional utilization of the perceptual loss (d) improves the level of detail.
with a NeRF. Neural Actor (NA) achieves high-quality results by utilizing the adversarial supervision in texture space. Neural Human Renderer (NHR) [59] optimizes features that are anchored to point clouds. We additionally compare to the concurrent work HDHumans [14], which is a hybrid method that combines the deformable human model [16] with a NeRF. Similar to NA, HDHumans also employs adversarial supervision in texture space. We show our comparison results on the  $D_{2}$  subject with tight clothing, as other baselines except DDC and HDHumans cannot handle the loose garment types. Note that we present both the qualitative and quantitative results in the order of non-real time, i.e., NB, A-NeRF, NA, HDHumans, and real-time methods, i.e., NHR, NV, DDC, and ours.
Fig. 4-(A) shows the comparison on the novel view synthesis task. Different from their original results [43], NB produces blurry results when trained on longer and more challenging training sequences, e.g., around 20k frames, as they fail to optimize the appearance code for every frame. A-NeRF exhibits the loss of details under highly articulated poses. While DDC and NA can generate coarse details, they still lack high-frequency details. On the other hand, our method can recover, both, the coarse and fine details and only achieves slightly worse synthesis quality than HDHumans while being significantly faster ( $\times 100$ ). In Tab. 1a, our method outperforms all the non-real time and real-time methods except HDHumans. However, we would like to highlight again that HDHumans has a runtime in the order of seconds per frame, which prevents it from being used in any real-time application. It is worth noting that ours with  $L_{1}$ -only supervision (see Tab. 2-c) already outperforms other real-time methods and even the non-real time method NA.
Fig. 4-(B) shows the comparisons on the novel pose synthesis task. Again, our method can recover the sharp and high-frequency details that are close to the ones of HDHumans, while most of the other approaches struggle with the given challenging pose. This is consistent with the quantitative results in Tab. 1b, where we achieve the second best performance among all the methods and the best performance among the real-time methods on the perception-based metrics LPIPS and FID. Due to the innate nature of pose to appearance task, i.e., the one-to-many mapping [3, 32], our method can generate realistic looking details that can deviate from the ground truth, which explains the lower performance in terms of PSNR in Tab. 1b.
We perform ablation studies on the novel view synthesis task on the  $D_{2}$  subject. To accurately access the contribution of each component, we train all the variants only with  $L_{1}$  loss if not stated otherwise.
Impact of Two-surface Parameterization. To study the efficacy of the two-surface parameterization, we train the single-surface model ('single-surface') where the MLP generates color conditioned only on a single inner surface intersection coordinates and feature. We also train a single-surface variant that is additionally conditioned on the viewing direction ('single-surface + viewing direction'). In Tab. 2, our two-surface model (c) outperforms the single-surface variants (a,b). Also, we would like to again stress that our  $L_{1}$ -only supervised two-surface model (c) already surpasses all of the real-time methods including DDC, and also outperforms the non-real time method NA, which is trained with ground truth texture-map supervision and adversarial supervision. This confirms that the
Coarse mesh
a. Single-surface
b. Two-surface
Figure 6: Ablation on the deformable template quality. All baselines trained only with  $L_{1}$  loss. When using a coarse mesh model the single-surface baseline (a) suffers from the geometric error of the model. In contrast, our two-surface representation (b) has the capability of fixing geometric errors and shows similar quality compared to the result when using the original mesh model (c).
Original mesh
c. Two-surface
(Ours)
Method	PSNR↑	LPIPS↓	FID↓
a. Single-surface	31.34	22.74	28.96
b. Single-surface + viewing direction	32.19	20.38	23.88
c. Two-surface with L1	33.27	18.51	23.58
d. Two-surface with L1 + Lperc (Ours)	33.30	12.85	8.69
e. Single-surface with coarse mesh	29.63	27.75	42.84
f. Two-surface with coarse mesh	32.03	22.00	31.43
Table 2: Ablations on novel view synthesis for the  $D_{2}$  subject. All baselines except (d) are trained with  $L_{1}$ -only supervision. Our two-surface parameterization outperforms single-surface variants and the perceptual supervision further improves visual quality. Further, our results when using a coarse mesh model are still plausible demonstrating the robustness of our method.
improvement originates from the carefully-designed two-surface model. In Fig. 5, We visually ablate the impact of each design choice. Although conditioning on the viewing direction (b) improves the quality, the single-surface models (a,b) are tightly bound to the underlying mesh. On the other hand, our two-surface models (c,d) go beyond the mesh and recover the real geometry.
Impact of Perceptual Supervision. Perceptual supervision improves the details (e.g., wrinkles) in Fig. 5-d, which is also confirmed by the better performance in the perceptual metrics in Tab. 2-d.
Impact of Deformable Template Quality. To demonstrate the robustness of our method, we train and evaluate our method with a smoothed version of the original mesh model in Fig. 6 and Tab. 2-e,f. The quantitative result outperforms real-time methods including DDC in terms of PSNR. This shows that even coarser geometry is sufficient for our representation. Moreover, the single-surface model (Fig. 6-a) reveals and suffers from geometric error when the coarse template is used. On the other hand, our two-surface representation (Fig. 6-b) has the capability of fixing geometric errors and shows similar visual quality to the result when the original mesh model is used (Fig. 6-c).
We presented DELIFFAS, an animatable representation that can synthesize high-quality images of the avatar under a user-controlled skeletal motion and viewpoint in real-time. At the technical core, our method utilizes a surface light field parameterized with two deformable surfaces computed from a deformable human model. Then, our light slab queries pose-dependent temporal normal map features together with the intersecting uv coordinates in order to render the pixel color with just a single MLP evaluation per pixel. This efficient rendering allows perceptual supervision on the entire image. Our experiments show that the proposed representation can synthesize high-quality renderings even under challenging poses, and outperforms the state-of-the-art real-time methods.
Acknowledgments. Christian Theobalt and Marc Habermann were supported by ERC Consolidator Grant 4DReply (770784). Lingjie Liu was supported by Lise Meitner Postdoctoral Fellowship. This work was partially supported by National Science Foundation Award 2107454.
A Appendix - Overview
This appendix is organized as follows: Sec. B shows additional results including video results, ablations on image-level and patch-level perceptual supervision, ablation study with and without perceptual supervision, ablations on the single surface with the same network capacity, results when using SMPL, comparison with traditional multi-view stereo method; Sec. C provides information regarding the reproducibility, which includes implementation details, training details, and runtime at inference; Sec. D discusses the performance changes according to the template mesh quality, and possible alternatives for the template mesh; Sec. E presents the societal impacts our work can have; Sec. F discusses the limitations of this work.
B.1 Video Results
Video results of free-viewpoint renderings, novel view and pose synthesis, and comparison with the state-of-the-art baselines on the DynaCap dataset [16] can be found at https://vcai.mpi-inf.mpg.de/projects/DELIFFAS. We compare our DELIFFAS with the non-real-time hybrid method Neural Actor [32], and real-time explicit mesh-based method Deep Dynamic Characters [16].
B.2 Ablation on Image-level and Patch-level Perceptual Supervision
To study the impact of our full image-level supervision, we train a variant with patch-level perceptual supervision. Here, we use a  $32 \times 32$  patch. In Tab. 3, our approach trained with full-image-level perceptual supervision (Tab. 3-c) outperforms the patch-level variant (Tab. 3-b) in terms of the perceptual metrics. In Fig. 7, we ablate the visual impact. Employing full-image-level supervision leads to better reconstruction of details (e.g., wrinkles). We would like to again highlight that image-level supervision with perceptual loss is possible thanks to the fast rendering speed of our method.
Figure 7: Ablation of patch-based and image-level perceptual supervision. Variants with the perceptual supervision (b,c) shows better details than the result with  $L_{1}$ -only supervision (a). Our final model which employs the full image-level perceptual supervision (c) outperforms the variant with patch-level supervision (b).
B.3 Ablation Study with and without Perceptual Supervision
We excluded the perceptual supervision from the ablation study in the main text (see Tab. 2, Fig. 5, and Fig. 6) to verify the effectiveness of our two-surface design. In Tab. 4 and Fig. 8, we show the ablation results with perceptual supervision. Similar to the results without perceptual supervision, our full model outperforms other variants.
Method	PSNR↑	LPIPS↓	FID↓
a. Image-level with L1	33.27	18.51	23.58
b. Patch-level with L1 + Lperc	33.24	16.54	18.96
c. Image-level with L1 + Lperc (Ours)	33.30	12.85	8.69
Table 3: Ablation on image-level and patch-level supervision. Applying the perceptual supervision on the entire image (c) leads to better results than the patch-level perceptual supervision (b) or  $L_{1}$ -only supervision.
Method	PSNR↑	LPIPS↓	FID↓
\( L_1 \)
a. Single-surface	31.34	22.74	28.96
b. Single-surface + viewing direction	32.19	20.38	23.88
c. Single-surface + two U-Net	31.47	15.43	19.99
d. Two-surface (Ours)	33.27	18.51	23.58
e. Single-surface with coarse mesh	29.63	27.75	42.84
f. Two-surface with coarse mesh	32.03	22.00	31.43
\( L_1 + L_{perc} \)
g. Single-surface	31.36	16.53	10.90
h. Single-surface + viewing direction	31.95	15.96	9.01
i. Single-surface + two U-Net	31.26	16.21	9.61
j. Two-surface (Ours)	33.30	12.85	8.69
k. Single-surface with coarse mesh	29.55	20.05	15.22
l. Two-surface with coarse mesh	32.03	16.13	12.08
Table 4: Ablations on novel view synthesis for the  $D_{2}$  subject with and without perceptual supervision. Our two-surface parameterization performs better than the single-surface variants and the perceptual supervision further boosts the visual quality. In addition, our results with coarse mesh model shows reasonable performance, which confirms the robustness of our method.
B.4 Ablations on the Single Surface with the Same Network Capacity
The number of total parameters is larger in the two-surface model than in the single-surface model as it includes two separate U-Nets for extracting feature map from each surface while the single-surface model only has a single U-Net. The U-Net and MLP architecture used in both single and two-surface models are the same. To verify that the performance boost originates from the two-surface design rather than the increase in network capacity, we additionally trained a "single-surface with two concatenated U-Net" variant (see Fig. 9, Tab. 4-c,i). Even though now the network size is the same, our two-surface representation still outperforms the single-surface baseline.
B.5 Using SMPL as the Deformable Mesh Surface
We conduct an experiment where we use the SMPL model as the deformable mesh model (see Fig. 10). One can see that our approach even for this very coarse model still outperforms the baselines proving the flexibility of our representation while a better template mesh further improves the results.
B.6 Comparison with Traditional Multi-view Stereo Method
We compare with the commercial photogrammetry software, Metashape [1] in Fig. 11. The reconstructed surface is noisy and the texture suffers from ghosting artifacts due to the inaccurate geometry. Also, Metashape takes 6 minutes to reconstruct while ours works in real-time (31fps). Lastly, our method is animatable, while multi-view stereo methods are not.
Figure 8: Ablations on novel view synthesis for the  $D_{2}$  subject with and without perceptual supervision. The proposed two-surface representation helps to recover the real geometry while the single-surface design is limited to the underlying mesh. Perceptual supervision further enhances the quality.
Figure 9: Ablations on the single surface with the same network capacity. Evaluated on the novel view synthesis task for the  $D_{2}$  subject. Even though the network size of the "single-surface with two U-Net" variant and ours are the same, our two-surface representation still outperforms the single-surface baseline. This again verifies that the performance boost is coming from the two-surface design, and not from using more network.
[  $\mathcal{L}_1$  -only]
$[\mathcal{L}_1 + \mathcal{L}_{perc}]$
Original mesh
Two-surface
[  $\mathcal{L}_1$  -only]
$[\mathcal{L}_1 + \mathcal{L}_{perc}]$
Figure 10: Ablations on using SMPL as a deformable human model. Evaluated on the novel view synthesis task for the  $D_{2}$  subject. Our method with very coarse SMPL mesh can still recover the real geometry compared to the single surface baseline where the approach can only paint on the given mesh.
C Reproducibility
Feature Extractor. The feature extractor for each surface is based on the same U-Net architecture [19]. We use the feature extractor to encode the temporal normal map  $(T + 1) \times 1024 \times 1024 \times 3$  into the  $1024 \times 1024 \times 32$  feature map. Here, the temporal normal map is the concatenation of the posed normal maps of the motion window  $[f, f - T]$ . The feature extractor consists of 10 down-sampling layers (with a down-sampling rate of two) and 10 up-sampling layers (with an up-sampling rate of two). There is a skip connection between every corresponding down and upsampling layer.
Light Field MLP. The light field MLP consists of 8 layers with 256 channel size. ReLU activation is used in between the layers. Input to the light field MLP are the positional encodings of two intersection uv coordinates. Then we concatenate the temporal normal map features sampled from the inner and outer surface to the intermediate output of  $4^{th}$  layer. Output of the light field MLP is the 3-channel RGB value.
Deformable Human Model. We leverage the deformable human model of Habermann et al. [16], i.e., DDC. The geometry networks of the DDC model are additionally trained with a Chamfer distance supervision with respect to 4D multi-view stereo reconstructions. Originally, we reported the numbers provided by the authors of DDC in Tab. 1. We trained TexNet of DDC with our exact checkpoint for
Metashape
GT
Figure 11: Comparison with MVS method (Metashape). We compare with the commercial photogrammetry software, Metashape [1]. The reconstructed surface is noisy and the texture exhibits ghosting artifacts. Also, Metashape takes 6 minutes to reconstruct while ours work in real-time (31fps). Lastly, our method is animatable, while multi-view stereo methods are not.
Table 5: We trained TexNet of DDC with our exact checkpoint for the geometry networks (EGNet and DeltaNet) that are trained with Chamfer distance to alleviate any other influence. We found that our geometry checkpoint has even a lower performance than the original one. Note that our representation clearly outperforms DDC.
Method	PSNR↑	LPIPS↓	FID↓
DDC (paper)	32.96	20.07	27.73
DDC (same geometry network)	29.50	32.66	44.18
Ours	33.30	12.85	8.69
(a) View synthesis error on D2
Method	PSNR↑	LPIPS↓	FID↓
DDC (paper)	28.05	30.43	38.37
DDC (same geometry network)	26.59	43.43	61.92
Ours	27.95	26.59	26.16
the geometry networks trained with Chamfer distance supervision (EGNet and DeltaNet) to alleviate any other influence. We found that our geometry checkpoint has lower performance than the original one. Note that our representation clearly outperforms DDC (see Tab. 5). Moreover, our ablation 'single surface + viewing direction' (Fig. 5-b,Tab. 2-b) can be considered as a variant of DDC further confirming the superiority of our appearance representation.
DELIFFAS. Both the feature extractors and light field MLP are trained in an end-to-end manner using Adam optimizer [25]. We used the learning rate of  $2 \times 10^{-4}$ . We train on a single RTX 8000 48G GPU with a single batch size. We supervise our approach by minimizing the following loss  $\mathcal{L} = \lambda_1 \cdot \mathcal{L}_1 + \lambda_{\mathrm{perc}} \cdot \mathcal{L}_{\mathrm{perc}}$ , where  $\mathcal{L}_1$  and  $\mathcal{L}_{\mathrm{perc}}$  are the  $L_1$  loss and perceptual [22] loss, respectively.  $\lambda_1$  and  $\lambda_{\mathrm{perc}}$  are their weights. For the first  $940k$  iterations,  $\lambda_1$  is set to one, and  $\lambda_{\mathrm{perc}}$  is set to zero. Then, we set  $\lambda_1$  and  $\lambda_{\mathrm{perc}}$  so that the magnitude of each loss term is roughly the same and train for additional  $350k$  iterations.
C.3 Runtime at Inference
We report our runtime on the novel pose synthesis task of  $D_{2}$  subject. DELIFFAS runs at 31 fps for rendering a single  $1K$  ( $940 \times 1285$ ) image on a single A100 GPU with an Intel Xeon CPU. The detailed breakdown of runtime is as follows.
Our pipeline can be divided into three stages: (1) obtaining UV (2) feature map generation (CNN) (3) MLP forwarding to get the final color value. (1) Obtaining UV: We utilize a GPU accelerated rasterizer to render the image space UV map, which takes  $3.58~\mathrm{ms}$ . (2) Feature map generation (CNN): Our U-Nets generate normal feature maps in  $18.85~\mathrm{ms}$ . (3) MLP forwarding to compute the final color: We take  $6.78\mathrm{ms}$  to bi-linearly sample the feature corresponding to each pixel, and then the light field MLP takes  $2.52\mathrm{ms}$  to compute the color value. In total, DELIFFAS takes  $31.73$  ms to render a single  $1K$  image.
Figure 12: Ablations on the deformable template quality with and without perceptual supervision. Evaluated on the novel view synthesis task for the  $D_{2}$  subject. Even when using the coarse mesh, our two-surface design enables to generate visual quality similar to the rendering result using original mesh. Employing the perceptual supervision further improves the quality.
Note that we assume the output of the deformable character model, i.e., DDC [16], is already available at training and inference time. DDC is also a real-time method so coupling DDC and our method would not deteriorate the performance much.
We would like to highlight that our method achieves the best of two worlds: runs in real-time as the explicit mesh-based method DDC (25fps), while at the same time achieving superior and comparable visual quality to the hybrid method NA (0.2fps) and HDHumans (0.3fps), respectively.
When tested on an A40 graphics card, we achieve a real-time speed of  $26fps$ . Furthermore, most of our computational budget is taken by the U-Net feature map extraction (  $60\%$  ) and deterministic bi-linear sampling of features  $(20\%)$ . We use a standard U-Net architecture and Tensorflow's bi-linear sampling, but more efficient architecture and sampling implementation can be explored to further reduce the runtime.
D Discussion on the Template Mesh Quality
**Rendering Quality.** Our performance degrades if a coarser mesh is used. However, note that our method with coarse geometry or even SMPL mesh can still recover the real geometry compared to the single surface baseline where the approach can only paint on the given mesh (see Fig. 6, Fig. 10, and Fig. 12). Also, it would be an interesting research direction if we could jointly refine the underlying mesh during training and improve the rendering quality.
Possible Alternatives to the Deformable Human Model. The deformable human model we used in the main text is DDC [16]. Alternatively, one can use the SMPL model (see Fig. 10) or SMPL+D to deal with the loose clothing. Also, we can use recent avatar NeRF works [64, 21, 42] to compute the canonical mesh (template mesh) and then use skinning-based deformation to obtain the deformed mesh for the new pose.
E Societal Impacts