SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Intuitively, applying supervision to different tri-plane combinations encourages the model to predict meaningful reconstruction in all three branches. To encourage smooth tri-plane reconstruction, we also adopt the TV loss proposed in [9]. The training objective for the first stage is  $\mathcal{L}^1 = \lambda_1\mathcal{L}_1 + \lambda_p\mathcal{L}_p + \lambda_{TV}\mathcal{L}_{TV} + \lambda_{neutral}\mathcal{L}_{neutral}$ , where  $\lambda_{x}$  is the weight of the corresponding objective.
In the second stage, to encourage multi-view consistency, we only fine-tune the super-resolution module and freeze other parts of the model. We use all the losses in the first stage and a dual-discriminator proposed in [9] that takes the concatenation of the low-resolution rendering and the high-resolution reconstruction as input. Specifically, we use the logistic formulation [22; 27; 58] of the adversarial loss  $L_{adv} = \mathbb{E}_{(I_o^l,I_o)}$  softplus  $(D(I_o^l\oplus I_o))$ , where  $I_o^l$  is the upsampled version of the low-resolution rendered image and  $\oplus$  represents the concatenation operation. The overall training objective of the second stage is  $\mathcal{L}^2 = \mathcal{L}^1 +\lambda_{adv}\mathcal{L}_{adv}$ .
Training datasets. We train our model using a single-view image dataset (FFHQ [26]) and two video datasets (CelebV-HQ [74] and RAVDESS [37]). For the single-view images in FFHQ, we carry out the 3D portrait reconstruction task, i.e., the source and target images are exactly the same. For the CelebV-HQ and the RAVDESS datasets, we randomly sample two frames from the same video to formulate a pair of images with the same identity but different motion. Furthermore, we observe that parts of videos in the CelebV-HQ and the RAVDESS datasets are fairly static, leading to a pair of source and target images with similar head poses, impeding the model from learning correct 3D shape of portraits. To enhance the learning of 3D reconstruction, we further employ an off-the-shelf 3D-aware GAN model [9] to synthesize 55,857 pairs of images rendered from two randomly sampled camera views, i.e., the source and target images have the same identity and expression but different views.
Evaluation datasets. We evaluate our method and the baselines [29; 24; 38; 65] on the CelebA dataset [32] and the testing split of the HDTF dataset [71], following [65; 38]. Note that our method has never seen any image from these two datasets during training while both StyleHeat[65] and OTAvatar [38] are trained using the training split of the HDTF dataset. Nonetheless, our method generalizes well to all validation datasets and achieves competitive performance, as discussed later.
Table 1: Comparison on CelebA [32].  ${}^{ \dagger  }$  Evaluated on a subset of CelebA,as discussed in Sec. 4.4.
Methods	3D Portrait Reconstruction	Cross-Identity Reeanct
L1↓	LPIPS↓	PSNR↑	SSIM↑	FID↓	CSIM↑	AED↓	APD↓	FID↓
ROME [29]	0.032	0.085	23.47	0.847	11.00	0.505	0.244	0.032	34.45
Ours	0.015	0.040	28.61	0.946	2.457	0.531	0.251	0.023	25.26
HeadNeRF†[24]	0.135	0.314	13.86	0.748	65.87	0.224	0.285	0.027	117.1
Ours†	0.024	0.098	25.83	0.883	9.400	0.591	0.278	0.017	22.97
Datasets pre-processing. We compute the camera poses of images in all training and testing datasets using [14] following [9]. As in previous literature [29; 24; 76], background modeling is out of the scope of this work; we further use an off-the-shelf portrait matting method [28] to remove backgrounds in all training and testing images.
4.2 Metrics and Baselines
We evaluate all methods for 3D portrait reconstruction, same-identity and cross-identity reenactment.
3D portrait reconstruction. We use all 29,954 high-fidelity images² in CelebA [32]. We measure the performance by computing various metrics between the reconstructed images and the input images, including the L1 distance, perceptual similarity metric (LPIPS), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and Fréchet inception distance (FID).
Same-identity reenactment. We follow [38] and use the testing split of HDTF [71], which includes 37,860 frames in total. We use the first frame of each video as the source image and the rest of the frames as target images. Following the evaluation protocol in [38; 65], we evaluate PSNR, SSIM, cosine similarity of the identity embedding (CSIM) based on [12], average expression distance (AED) and average pose distance (APD) based on [14], average keypoint distance (AKD) based on [8], as well as LPIPS, L1 and FID between the reenacted and ground truth frames.
Cross-identity reenactment. We conduct cross-identity reenactment on both the CelebA and HDTF datasets. For CelebA, we split the dataset into 14,977 image pairs and transfer the expression and head pose from one image to the other. As for the HDTF dataset, we follow [38] and use one clip as the driving video and the first frame of the other videos as source images, which produces 67,203 synthesized images in total. To fully evaluate the performance of one-shot avatar animation, we transfer motion from the HDTF videos to the single-view images in the CelebA dataset. Similar to [65], we use the first 100 frames in the HDTF videos as target images and 60 images sampled from the CelebA as source images, resulting in a total of 114,000 synthesized images. Since there is no ground truth for cross-identity reenactment, we evaluate the results based on the CSIM, AED, APD, and FID metrics. More evaluation details can be found in the supplementary.
Baselines. In terms of baselines, we compare our method against a SOTA 2D talking head synthesis method [65], two SOTA 3D head avatar animation methods [24; 29], and one concurrent work [38].
We implement the proposed method using the PyTorch framework [45] and train it with 8 32GB V100 GPUs. The first training stage takes 6 days, consisting of 750000 iterations. The second training stage takes 2 days with 75000 iterations. Both stages use a batch size of 8 with the Adam optimizer [30] and a learning rate of 0.0001. More implementation details can be found in the supplementary.
4.4 Qualitative and Quantitative Results
3D portrait reconstruction. For each testing portrait image, we reconstruct its 3D head avatar and render it from the observed view using different methods. By comparing the rendering with the input image, we assess the fidelity of 3D reconstruction of each method. Table 1 shows the quantitative results, demonstrating that our model achieves significantly better reconstruction and fidelity scores. These results highlight the ability of our model to faithfully capture details in the input images and reconstruct high-fidelity 3D head avatars. Visual examples are present in the supplementary.
$^{2}$ Due to the time-consuming nature of HeadNeRF, we compare our method with HeadNeRF on a subset of 3000 images from the CelebA dataset.
Figure 3: Cross-identity reenactment on CelebA [32] and HDFT [71]. The first two rows show cross-identity reenactment results on the CelebA dataset, while the last two rows demonstrate motion transfer from videos in the HDFT dataset to images in the CelebA dataset.
Table 2: Comparison on the HDTF dataset [71].
Methods	Same-Identity Reenactment	Cross-Identity Reenactment
PSNR↑	SSIM↑	CSIM↑	AED↓	APD↓	AKD↓	LPIPS↓	L1↓	FID↓	CSIM↑	AED↓	APD↓	FID↓
ROME [29]	20.75	0.838	0.746	0.123	0.012	2.938	0.173	0.047	31.55	0.629	0.247	0.020	43.38
OTAvatar [38]	20.12	0.806	0.619	0.162	0.017	2.933	0.198	0.053	36.63	0.514	0.282	0.028	44.86
StyleHeat [65]	19.18	0.805	0.654	0.141	0.021	2.843	0.194	0.056	108.3	0.537	0.246	0.025	105.1
Ours	22.15	0.868	0.789	0.129	0.010	2.596	0.117	0.037	21.60	0.643	0.263	0.018	47.39
Cross-identity reenactment. Fig. 3, and Fig. 4 showcase the qualitative results of cross-identity reenactment on the CelebA [32] and HDFT [71] dataset. Compared to the baselines [29; 65; 38], our method faithfully reconstructs intricate details such as hairstyles,
earrings, eye glasses etc. in the input portrait images. Moreover, our method successfully synthesizes realistic appearance change corresponding to the target motion. For instance, our model is able to synthesize plausible teeth when the mouth transitions from closed to open (e.g., row 3 in Fig. 3), it also hallucinates the occluded face region when the input image is in profile view (e.g., row 2 in Fig. 3). In contrast, the mesh-based baseline [29] can neither capture photo-realistic details nor hallucinate plausible inner mouths, while the 2D talking head synthesis baseline [65] produces unrealistic warping artifacts when the input portrait is in the profile view (e.g. row 2 in Fig. 3). We provide quantitative evaluations of the cross-identity reenactment results in Table 1, Table 2, and Table 3. Our method demonstrates better fidelity and identity preservation scores, showing its strong ability in realistic portrait synthesis. It is worth noting that HDTF [71] includes images that are less sharp compared to the high-fidelity images our model is trained on, which may account for the slightly lower FID score in Table 2.
Table 3: Cross-identity reenactment between the HDTF dataset [71] and the CelebA dataset [32].
Methods	CSIM↑	AED↓	APD↓	FID↓
ROME [29]	0.521	0.270	0.022	76.03
StyleHeat [65]	0.461	0.270	0.038	94.28
Ours	0.551	0.274	0.017	59.48
Figure 5: Ablation studies. Details are explained in Sec. 4.5.
Table 4: Ablation studies. Blue text highlights the inferior performance of the variants. (Sec. 4.5)
Methods	3D Portrait Reconstruction	Cross-Identity Reeanct
L1↓	LPIPS↓	PSNR↑	SSIM↑	FID↓	CSIM↑	AED↓	APD↓	FID↓
w/o perspective	0.061	0.239	19.53	0.712	47.84	0.370	0.243	0.015	43.33
w/o neutral constraint	0.027	0.124	25.65	0.854	13.56	0.593	0.315	0.018	22.92
linear expression	0.031	0.134	25.08	0.841	14.62	0.443	0.217	0.016	26.18
Ours	0.030	0.116	24.77	0.861	10.47	0.599	0.276	0.017	17.36
Same-identity reenactment. Table 2 shows the quantitative results of same-identity reenactment on HDTF [71]. Our method generalizes well to HDTF and achieves better metrics compared to existing SOTA methods [29; 38; 65]. The qualitative results of same-identity reenactment can be found in the supplementary.
Figure 4: Cross-identity reenactment on HDFT [71].
Efficiency. Since HeadNeRF [24]
and OTAvatar [38] require latent code optimization for unseen identities, the process of reconstructing and animating an avatar takes them 53.0s and 19.4s, respectively. Meanwhile, ROME [29] and our method only needs an efficient forward pass of the network for unseen identities, taking 1.2s and 0.6s respectively. Overall, our method strikes the best balance in terms of speed and quality.
We conduct experiments to validate the effectiveness of the neutral expression constraint (Sec. 3.1), the contribution of the appearance branch (Sec. 3.2), and the design of the expression branch (Sec. 3.3).
Neutral expression constraint. In our model, we aim to wipe out the expression in the source image by enforcing the coarse reconstruction from the canonical branch to have a neutral expression. This ensures that the expression branch always animates a fully "neutralized" expressionless face from the canonical branch. Without this, the expression branch fails to correctly modify the coarse reconstruction into the target expression, as shown in Fig. 5(e) and Table 4 (i.e., worse AED score).
Appearance branch. The appearance branch is the key to reconstructing intricate facial details of the input portrait image. Without this branch, the model struggles to capture photo-realistic details, resulting in considerably lower reconstruction and fidelity metrics, as shown in Table 4 and Fig. 5(f).
Alternative expression branch design. Instead of using the frontal view rendering from the 3DMM to provide target expression information to the expression branch (see Sec. 3.3), an alternative way is to use the 3DMM expression coefficients to linearly combine a set of learnable expression bases. However, this design performs sub-optimally as it overlooks the individual local deformation details caused by expression changes, introducing artifacts (e.g., mouth not fully closed) shown in Fig. 5(d) and lower FID in Table 4. We provide more results and ablations in the supplementary.
In this paper, we propose a framework for one-shot 3D human avatar reconstruction and animation from a single-view image. Our method excels at capturing photo-realistic details in the input portrait image, while simultaneously generalizing to unseen images without the need for test-time optimization. Through comprehensive experiments and evaluations on validation datasets, we demonstrate that the proposed approach achieves favorable performance against state-of-the-art baselines on head avatar reconstruction and animation.
[61] Eric Zhongcong Xu, Jianfeng Zhang, Junhao Liew, Wenqing Zhang, Song Bai, Jiashi Feng, and Mike Zheng Shou. Pv3d: A 3d generative model for portrait video generation. In ICLR, 2023. 3
[62] Hongyi Xu, Guoxian Song, Zihang Jiang, Jianfeng Zhang, Yichun Shi, Jing Liu, Wanchun Ma, Jiashi Feng, and Linjie Luo. Omniavatar: Geometry-guided controllable 3d head synthesis. arXiv preprint arXiv:2303.15539, 2023. 3
[63] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. arXiv preprint arXiv:2201.08845, 2022. 5
[64] Yiran Xu, Zhixin Shu, Cameron Smith, Jia-Bin Huang, and Seoung Wug Oh. In-n-out: Face video inversion and editing with volumetric decomposition. arXiv preprint arXiv:2302.04871, 2023. 3
[65] Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In ECCV, 2022. 1, 2, 6, 7, 8, 9
[66] Ziyang Yuan, Yiming Zhu, Yu Li, Hongyu Liu, and Chun Yuan. Make encoder great again in 3d gan inversion through geometry and occlusion-aware encoding. arXiv preprint arXiv:2303.12326, 2023. 3, 4
[67] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. Fast bi-layer neural synthesis of one-shot realistic head avatars. In ECCV, 2020. 1, 2
[68] Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Fdnerf: Few-shot dynamic neural radiance fields for face reconstruction and expression editing. In SIGGRAPH Asia, 2022. 3
[69] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 4
[70] Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. arXiv preprint arXiv:2211.12194, 2022. 1, 2
[71] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset. In CVPR, 2021. 6, 7, 8, 9
[72] Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, and Otmar Hilliges. I M Avator: Implicit morphable head avatars from videos. In CVPR, 2022. 2, 3
[73] Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J. Black, and Otmar Hilliges. Pointavatar: Deformable point-based head avatars from videos. In CVPR, 2023. 3
[74] Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. CelebV-HQ: A large-scale video facial attributes dataset. In ECCV, 2022. 6
[75] Peiye Zhuang, Liqian Ma, Sanmi Koyejo, and Alexander Schwing. Controllable radiance fields for dynamic face synthesis. In 3DV, 2022. 3
[76] Yiyu Zhuang, Hao Zhu, Xusen Sun, and Xun Cao. Mofanerf: Morphable facial neural radiance field. In ECCV, 2022. 3, 7