Generalizable One-shot 3D Neural Head Avatar
Authors: Xueting Li
Shalini De Mello
Sifei Liu
Koki Nagano
Umar Iqbal
Jan Kautz
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural Radiance Field
Portrait Reconstruction and Animation
TL;DR: In this paper, we present a method that reconstructs and animates any 3D head avatar from a single-view portrait image.

Abstract: We present a method that reconstructs and animates a 3D head avatar from a single-view portrait image. Existing methods either involve time-consuming optimization for a specific person with multiple images, or they struggle to synthesize intricate appearance details beyond the facial region. To address these limitations, we propose a framework that not only generalizes to unseen identities based on a single-view image without requiring person-specific optimization, but also captures characteristic details within and beyond the face area (e.g. hairstyle, accessories, etc.). At the core of our method are three branches that produce three tri-planes representing the coarse 3D geometry, detailed appearance of a source image, as well as the expression of a target image. By applying volumetric rendering to the combination of the three tri-planes followed by a super-resolution module, our method yields a high fidelity image of the desired identity, expression and pose. Once trained, our model enables efficient 3D head avatar reconstruction and animation via a single forward pass through a network. Experiments show that the proposed approach generalizes well to unseen validation datasets, surpassing SOTA baseline methods by a large margin on head avatar reconstruction and animation.

Generalizable One-shot 3D Neural Head Avatar
Xueting Li, Shalini De Mello, Sifei Liu, Koki Nagano, Umar Iqbal, Jan Kautz
NVIDIA
avataa
Abstract
We present method that reconstructs and animates 3D head avatar from single- view portrait image. Existing methods either involve time-consuming optimization for specific person with multiple images, or they struggle to synthesize intricate appearance details beyond the facial region. To address these limitations, we propose framework that not only generalizes to unseen identities based on a single-view image without requiring person-specific optimization, but also captures characteristic details within and beyond the face area (e.g. hairstyle, accessories, etc.). At the core of our method are three branches that produce three tri-planes representing the coarse 3D geometry, detailed appearance of source image, well as the expression of a target image. By applying volumetric rendering to the combination of the three tri-planes followed by super- resolttino module, our method yields a high fidelity image of the desired identity, expression and pose. Once trained, our model enables efficient 3D head avatar reconstruction and animation via a single forward pass through a network. Experiments show that the proposed approach generalizes well to unseen validation datasets, surpassing SOTA baseline methods by a large margin on head avatar reconstruction and animation.
Introduction
Head avatar animation [57; 29; 38] aims to animate source portrait image with the motion (i.e., pose and expression) from a target image. It is a long-standing task in computer vision that has been widely applied to video conferencing, computer games, Virtual Reality (VR) and Augmented Reality (AR). real-world applications, synthesizing a realistic portrait image that matches the given identity and motion raises two major challenges efficiiency and high fidelity. Efficiency requires the model to generalize to arbitrary unseen identities and motion without any further optimization during inference. High fidelity demands the model not only faithfully preserve intricate details in the input image (e.g. hairstyle, glasses, earrings), but also hallucinate plausibly whenever necessary (e.g. synthesize the occluded facial region when the input is in profile view or generate teeth when the mouth transitions from closed to open).
Traditional methods [14; 17; 11] based 3D Morphable Models (3DMMs) learn networks that predict shape, expression, pose and texture of an arbitrary source portrait image efficiently. However, these approaches often fall short in synthesizing realistic details due limited mesh resolution and a coarse texture model. Additionally, they exclusively focus on the facial region while neglecting other personal characteristics such as hairstyle glasses. Inspired by the remarkable progress made in Generative Adversarial Networks (GANs) [22; 27; 58], another line of methods [57; 65; 67; 48; 46; 70; 15] represent motion as warping field that transforms the given source image to match the desired pose and expression. Yet, without explicit 3D understanding of the given portrait image, these methods can only rotate the head within limited angles, before exhibiting warping artifacts, unrealistic distortions and undesired identity changes across different target views. Recently, neural rendering [39] has demonstrated impressive results in facial avatar reconstruction and
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

animation [19; 42; 54; 55; 72; 2; 43; 20; 21; 23; 4]. Compared to meshes with fixed and pre-defined topology, an implicit volumetric representation capable of learning photo-realistic details including areas beyond the facial region. However, these models have limited capacity and cannot generalize trivially to unseen identities during inference. As result, they require time-consuming optimization and extensive training data of a specific person to faithfully reconstruct their 3D neural avatars.
In this paper, we present a framework aiming at a more practical but challenging scenario - given an unseen single-view portrait image, we reconstruct an implicit 3D head avatar that not only captures photo-realisn details within and beyond the face region, but also is readily available for animation without requiring further optimization during inference. To this end, we propose framework with three branches that disentangle and reconstruct the coarse geometry, detailed appearance and expression of a portrait image, respectively. Specifically, given a source portrait image, our canonical branch reconstructs its coarse 3D geometry by producing a canonicalized tri-plane [9; 10] with neutral expression and frontal pose. To capture the fine texture and characteristic details of the input image, we introduce an appearance branch that utilizes the depth rendered from the canonical branch to create a second tri-plane by mapping pixel values from the input image onto corresponding positions the canonicalized 3D space. Finally, we develop an expression branch that takes the frontal rendering of 3DMM with a target expression and a source identity as input. It then produces third tri-plane that modifies the expression of the reconstruction as desired. After combining all three tri tri-planes by summation, produce a high-fidelity facial image with source identity as well as target pose and expression. Our model is learned with large numbers of portrait images of various identity and motion during training. At inference time, it can be readily applied to an unseen single-view image for 3D reconstruction and animation, eliminating the need for additional test-time optimization.
carry out volumetric rendering followed by super-resolution block and
To summarize, our contributions are:
We propose a framework for 3D head avatar reconstruction and animation that simultane- ously captures intricate details in a portrait image while generalizing to unseen identities without test-time optimization.
To achieve this, we introduce three novel modules for coarse geometry, detailed appearance well as expression disentanglement and modeling, respectively.
*
Our model can be directly applied to animate an unseen image during inference efficiently, achieving favourable performance against state-of-the-art head avatar animation methods.
2 Related Works
2.1 3D Morphable Models
Reconstructing and animating 3D faces from images has been a fundamental task in computer vision. Following the seminal work by Parke el al. [44], numerous methods have been proposed to represent the shape and motion of human faces by 3D Morphable Models (3DMMs) [1; 36; 16; 7; 35]. These methods represent the shape, expression and texture of given person by linearly combining a set of bases using person-specifi parameters. Building upon 3DMMs, many works have been proposed to reconstruct and animate human faces by estimating the person-specific parameters given single-view portrait image [14; 17; 11; 34]. While 3DMMS provide a strong prior for understanding of human faces, they are inherently limited in two ways. First, they exclusively focus on the facial region and fail to capture other characteristic details such as hairstyle, eye glasses, inner mouth etc. Second, the geometry and texture fidelity of the reconstructed 3D faces are limited by mesh resolution, leading to unrealistic appearance in the rendered images. In this work, we present a method that effectively exploits the strong prior 3DMMs while addressing its geometry and texture fidelity limitation by employing neural radiance fields [39; 5; 41].
2.2 2D Expression Transfer
The impressive performance of Generative Adversarial Networks (GANs) [22] spurred another line of head avatar animation methods [57; 65; 67; 48; 46; 70; 15]. Instead of reconstructing the underline 3D shape of human faces, these methods represent motion (i.e. expression and pose) as a warping field. Expression transfer is carried out by applying warping operation onto the source image to match the motion of the driving image. By leveraging the powerful capacity of generative models, these methods produce high fidelity results with more realistic appearance compared to 3DMM-based
2

---

methods. However, without an explicit understanding and modeling of the underlying 3D geometry of human faces, these methods usually suffer from warping artifacts. unrealistic distortions and undesired identity change when the target pose and expression are significantly different from the ones in the source image. contrast, we explicitly reconstruct the underline 3D geometry and texture of a portrait image, enabling our method to produce more realistic synthesis even in cases of large pose change during animation.
2.3 Neural Head Avatars
Neural Radiance Field (NeRF) [39; 41] debuts remarkable performance for 3D scene reconstruction. Many works [19; 42; 54; 55; 72; 2; 43; 20; 21; 23; 4; 31; 73; 3] attempt to apply NeRF to human portrait reconstruction and animation by extending it from static scenes to dynamic portrait videos. Although these methods demonstrate realistic reconstruction results, they inefficiently learn separate networks for different identities and require thousands of frames from specific individual for training. Another line of works focus on generating controllable 3D head avatar from random noise [53; 61; 51; 40; 62; 33; 75; 50]. Intuitively, 3D face reconstruction and animation could be achieved by combining these generative methods with GAN inversion [47; 18; 64; 60J. However, the individual optimization process for each frame during GAN inversion is computationally infeasible for real-time performance in applications such as video conferencing. Mcanwhile, several works [56; 66; 6; 13] focus on reconstructing 3D avatars from arbitrary input images, but they cannot animate or reenact these avatars. Closest to our problem setting, few works explore portrait reconstruction and animation in a few-shot [68] or one-shot [24; 29; 38; 76] manner. Specifically, the ROME method [29] combines a learnable neural texture with explicit FLAME meshes [36] to reconstruct 3D head avatar, encompassing areas beyond the face region. However, using meshes as the 3D shape representation prevents the model from producing high-fidelity geometry and appearance details. Instead of using explicit meshes as 3D representation, the HeadNeRF [24] and MofaNeRF methods learn implicit neural networks that take 3DMM parameters (i.e. identity and expression coefficients or albedo and illumination parameters) as inputs to predict the density and color for each queried 3D point. Additionally, the OTAvatar [38] method proposes to disentangle latent style codes from pre-trained 3D-aware GAN [9] into separate motion and identity codes, enabling facial animation by exchanging the motion codes. Nonetheless, all three models [24; 76; 38] require laborious test-time optimization, and struggle to reconstruct rhoto-realistic texture details of the given portrait image presumably because they encode the appearance using compact latent vector. In this paper, propose the first 3D head neural avatar animation work that not only generalizes to unseen identities without test-time optimization, but also captures intricate details from the given portrait image, surpassing all previous works in quality.
3 Method
We present a framework that takes a source image 1, together with a target image L as inputs, and synthesizes an image I, that combines the identity from the source image and the motion (i.e., expression and head pose) from the target image. The overview of the proposed method is illustrated in Fig. 1. Given a source image including human portrait, we begin by reconstructing the coarse geometry and fine-grained person-specific details via canonical branch and an appearance branch, respectively. To align this reconstructed 3D neural avatar with the expression in the target image, we employ an off-the-shelf 3DMM [14] model to produce frontal-view rendering that combines the identity from the source image with the expression from the target image. Our expression branch then takes this frontal- -iew rendering as input and outputs tri-plane that aligns the reconstructed 3D avatar to the target expression. By performing volumetric rendering from the target camera view and applying super-resolution block, we synthesize a high-fidelity image with the desired identity and motion. In the following, we describe the details of each branch in our model, focusing on answering three questions: a) how to reconstruct the coarse shape and texture of a portrait image with neutral expression in Sec. 3.1; b) how to capture appearance details in the source image in Sec. 3.2; and e) how to model and transfer expression from the target image onto the source image in Sec. 3.3. The super-resolution module and the training stages with associated objectives will be discussed in Sec. 3.4 and Sec. 3.5, respectively.
We introduce the preliminary of the 3DMM in the supplementary.
3

---

point
Lifting
Rasterization
image
source image I,
T
appearance branch
D
Ry
resolution
cancnical
3DMM
(b) Lifting
Tosp
Tg
branch
neural point cloud
reconstruction
tripllane
(a) Framework Overview
Rasterization
Figure 1: Overview. The proposed method contains four main modules: canonical branch that reconstructs the coarse geometry and texture of portrait with neutral expression (Sec. 3.1), an appearance branch that captures tne-grained person person-specific details (Sec. 3.2), an expression branch that modifies the reconstruction to desired expression, and super-resolution block that renders high-fidelity synthesis (Sec. 3.4).
3.1 Coarse Reconstruction via the Canonical Branch
Given source image L, depicting human portrait captured from the camera view C.. the canonical branch predicts its coarse 3D reconstruction represented as tri-plane [9; 10] Te. To serve strong geometric prior for the subsequent detailed appearance and expression modeling, we impose two crucial properties the coarse reconstruction. First, the coarse reconstruction of face images captured from different camera views should be aligned in the 3D canonical space, allowing the model generalize single-view portrait images captured from arbitrary camera views. Second, enforce the coarse reconstruction to have a neutral expression (i.e., opened eyes and closed mouth), which facilitates the expression branch add the target expression effectively.
Based on these two goals, we design an encoder Ee that takes the source image L. E 33x512x512 input and predicts canonicalized tri-plane T. E 323 a pre-trained SegFormer [59] model as our encoder, whose transformer design enables effective mapping from the 2D input the canonicalized 3D space. Furthermore, ensure that T. has neutral expression, we employ 3DMM [14] to render a face with the same identity and camera pose of the source image, but with neutral expression. We then encourage the rendering of T. be close to the 3DMM's rendering within the facial region by computing an LI loss and a perceptual loss [25; 69] between them:
Specifically, fine-tune
Ie Rv(Te, C.)
Ineus Mneu = Rm(as, Bo, C.) Cneutral Ineu Ie x Mnsull + [[@(Ineu) :(Ie x Mnew)ll,
where (T, C) is the volumetric rendering a tri-plane T from the camera view C, pre-trained VGG-19 network [49]. T, M Rm(a, 3, C.) is the 3DMM [14] that takes identity coefficients a, expression coefficients 3 as inputs, and renders an image I and a mask M including only the facial region from camera view C. By setting a a, (i.e. the identity coefficents of 1,) and 30 = in Eq. 1, we ensure that Lne has the same identity L, but with a neutral expression.
As shown in Fig. 2(c), the rendered image I from the canonical tri-plane T, indeed has neutral expression with opened eyes and closed mouth, but lacks fine-grained appearance. This is because mapping portrait from the 2D input to the canonicalized 3D space is a challenging and holistic process. As a result, the encoder primarily focuses on aligning inputs from different camera views and neglects individual appearance details. Similar observations have also been noted in [13; 66). To resolve this issue, we introduce an appearance branch that spatially transfers details from the input image to the learned coarse reconstruction's 's surface in the next section.

---

image 4
(b) target image 4
(c)
[0 + 7.)
(eIRg(T +T))
Figure 2: Visualization of the contribution of each branch. (a) Source image. (b) Target image. (c) Rendering of the canonical tri-plane. (d) Rendering of the combination of the canonical and expression tri-planes. (e) Rendering of the combination of all three tri-planes.
3.2 Detail Reconstruction via the Appearance Branch
We now introduce the appearance branch that aims to capture and reconstruct intricate facial details in the input image. The core idea is to leverage the depth map rendered from the canonical tri-plane Te compute the 3D position of each pixel in the image such that the facial details can be accurately "transferred" from the 2D input image to the 3D reconstruction, Specifically, we first render T. from the source camera view C, to obtain a depth image D. E R 128x122 128 The 3D position (denoted as pe) of each pixel I in the source image I, can be computed by p = 0 0- D d, where and d are the ray origin and viewing direction sampled from the camera view C. of the source image. Based on the 3D locations of all pixels, we construct a neural point cloud [63; 52] by associating the color information from each pixel I in the 2D image to its corresponding 3D position pis Instead of directly using the RGB color of each pixel, we employ an encoder E, to extract 2D features (denoted as F 32x x28x 128) from I, and associate the feature at each pixel to its corresponding 3D location. As result, we establish a neural point cloud composed of all visible pixels in the image and associate each point with a 32-dimensional feature vector. This mapping process from a 2D image to the 3D space, is referred to as "Lifting" and demonstrated in Fig. 1(b).
To integrate the neural point cloud into the canonical tri-plane Tr, we propose a "Rasterization" process (see Fig. I(c) that converts the neural point cloud to another tri- tri-plane denoted as T, such that it can be directly added to Te. For each location on the planes (i.e. the XY-, YZ-, XZ-plane) in Tn, we compute its nearest point in the neural point cloud and transfer the feature from the nearest point onto the query location on the plane. A comparison between Fig. 2(d) and Fig. 2(e) reveals the contribution of our appearance tri-plane Tp. which effectively transfers the fine- mne-grained details (e.g.. pattern on the hat) from the image onto the 3D reconstruction.
3.3 Expression Modeling via the Expression Branch
Expression reconstruction and transfer is challenging task. Naively predicting the expression from an image poses difficulties in disentangling identity, expression, and head rotation. Meanwhile, 3DMMs provide well-established expression representation that captures common human expressions effectively. However, the compact expression coefficients in 3DMMs are highly correlated with the expression bases and do not include spatially varying deformation details. As a result, conditioning a network solely on these coefficients for expression modeling can be challenging. Instead, we propose simple expression branch that fully leverages the expression prior any 3DMM and seamlessly integrates with the other two branches. The core idea is to provide the model with target expression information using a 2D rendering from the 3DMM instead of the expression coefficients. As shown in Fig. (a)), given the source image I, and target image I, we predict their corresponding shape and expression coefficients denoted as a', and 33 respectively using 3DMM prediction network [14]. By combining a, and 3t, we render a frontul-view facial image as Texp RM,,,,, 31, Croont), where C front pre-defined frontal camera pose. We then use an encoder (denoted as F.,) that takes lemp as input and produces an expression tri-plane T. E X83x3225662366 We modify the canonical tri-plane T, to the target expression by directly adding T. with T.. Note that we always render Teep in the pre-defined frontal view so that the expression encoder can focus on modeling expression changes only and ignore motion changes caused by head rotation. Moreover, our expression encoder also learns to hallucinate realistic inner mouths (e.g., teeth) according to the target expression, as the 3DMM rendering Teap does not model the inner mouth region. Fig. 2(d) visualizes the images rendered by combining the canonical and expression tri-planes, where the target expression from Fig, 2(b) is effectively transferred onto Fig. 2(a) through the expression tri-plane.
5

---

3,4 The Super-resolution Module
By adding the canonical and appearance tri-planes from a source image, together with the expression tri-plane from target image, We reconstruct and modify the portrait in the source image to match the target expression. Through volumetric rendering, we can obtain portrait image at a desired camera view. However, the high memory and computational cost of volumetric rendering prevents the model from synthesizing high-resolution output. To overcome this challenge, existing works [9; 33; 50; 51] utilize a super-resolution module that takes a low-resolution rendered image or feature map input and synthesizes high-resolution result. In this work, we follow this line of works and fine-tune a pre-trained GFPGAN [58; 51] as our super-resolution module [51]. By pre-training on the task of facial image super-resolution.
2D face restoration, GFPGAN learns strong prior for Additionally, its layer-wise Seature-condition design prevents the model from deviating from the low-resolution input, thereby mitigating temporal or multi-view inconsistencies, as observed in [51].
3.5 Model Training
We utilize two-stage training schedule to promote multi-view consistent reconstructions, well as to reduce the overall training time. In the first stage, we train our model without the super- resolution module using reconstruction objective and the neutral expression loss discussed in Sec. 3.1. Specifically, we compare the rendering of a) the canonical tri-plane (i.e., I = RyRRT C.)), b) the combination of the canonical and expression tri-planes (i.e., lote RRy + Te. C,)), and c) the combination of all three tri-planes (i.., lotetp = Ry(Te + Te + Tp, Ce) with the target image via the LI and the perceptual losses similarly Eq. 1:
L = || Tell + 1|Tete Tell + ITTtetp - Tell: C, = [l)(1)) - <(4))l 110(1++)))==)))))) (1a1ll + [0(Teto)p) O(1111-
Intuitively, applying supervision to different tri-plane combinations encourages the model predict meaningful reconstruction in all three branches. To encourage smooth tri-plane reconstruction, we also adopt the TV loss proposed in [9]. The training objective for the first stage is L X1C1+ + XpCp + Arv Crv + Aneutral Cneutral, Queutrai where Ag is the weight of the corresponding objective.
In the second stage, to encourage multi-viww consistency, we only fine-tune the super-resolution module and freeze other parts of the model. We use all the losses in the first stage and a dual- discriminator proposed in [9] that takes the concatenation of the low-resolution rendering and the high-resolution reconstruction as input. Specifically, we use the logistic formulation [22; 27; 58) of the adversarial loss Lade = (g sootsssofpuuustt low-resolution rendered image and @ represents the concatenation operation. The overall training objective of the second stage C L² + Aado
@ I,))), where I is the upsampled version of the
4 Experiments
4.1 Datasets
Training datasets. We train our model using a single-view image dataset (FFHQ (26]) and two video datasets (CelebV-HQ [74] and RAVDESS [37]). For the single-view images in FFHQ, carry out the 3D portrait reconstruction task, i.e., the source and target images are exactly the same. For the Ce and the RAVDESS datasets, we randomly sample two frames from the same video to formulate a pair of images with the same identity but different motion. Furthermore, we observe that parts of videos in the CelebV-HQ and the RAVDESS datasets are fairly static, leading to a pair of source and target images with similar head poses, impeding the model from learning correct 3D shape of portraits. To enhance the learning of 3D reconstruction, we further employ an off- tff-thh-shelf 3D 3D-aware GAN model [9] to synthesize 55,857 pairs of images rendered from two randomly sampled camera views, i.e., the source and target images have the same identity and expression but different views.
Evaluation datasets. We evaluate our method and the baselines [29; 24; 38; 65] on the CelebA dataset [32] and the testing split the HDTF dataset [71], following [65; 38]. Note that our method has never seen any image from these two datasets during training while both StyleHeat[65] and OTAvvatar [38] are trained using the training split of the HDTF dataset. Nonetheless, our method generalizes well to all validation datasets and achieves competitive performance, discussed later.

---

Table 1: Comparison on CelebA [32]." Evaluated on subset of CelebA, as discussed in Sec. 4.4.
3D Portrait Reconstruction
Cross-Identne Recanct
Methods
L11
LPIPS|
PSNRT
SSIMF
FID4
CSIM
AED
APD4
FID]
ROME [29]
0.032
0.085
23.47
0.847
0.505
0.244
0.032
34.45
Ours
0,015
0.040
28.61
0,946
2.457
0.531
0.251
0.023
25.26
HeadNeRFf [24]
0.135
0.314
13.86
0.748
65.87
0.224
0.285
0.027
117.1
Ours! 0.024
0.098
25.83
0.883
9.400
0.591
0.278
0.017
22.97
Datasets pre-processing. We compute the camera poses of images in all training and testing
datasets using [14] following [9]. As in previous literature [29; 24; 76]. background modeling is out of the scope of this work; we further use an off-the-shelf portrait matting method [28] remove backgrounds in all training and testing images.
4.2 Metrics and Baselines
We evaluate all methods for 3D portrait reconstruction, same-identity and cross-identity reenactment.
3D portrait reconstruction. We use all 29,954 high-fidelity images² in CelehA [32]. We measure the performance by computing various metrics between the reconstructed images and the input images, including the L1 distance, perceptual similarity metric (LPIPS), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and Fréchet inception distance (FID).
Same-identity reenactment. We follow [38] and use the testing split of HDTF [71], which includes 37,860 frames in total. We use the first frame of each video as the source image and the rest of the frames as target images. Following the evaluation protocol in [38; 65], we evaluate PSNR, SSIM, cosine similarity of the identity embedding (CSIM) based on [12], average expression distance (AHD) and average pose distance (APD) based on [14], average keypoint distance (AKD) based on [8], as well as LPIPS, L1 and FID between the reenacted and ground truth frames.
Cross-identity reenactment. We conduct cross-identity reenactment on both the CelebA and HDTF datasets. For CelebA, we split the dataset into 14,977 image pairs and transfer the expression and head pose from one image to the other. As for the HDTF dataset, we follow [38] and use one clip the driving video and the first frame of the other videos as source images, which produces 67,203 synthesized images in total. To fully evaluate the performance of one-shot avatar animation, transfer motion from the HDTF videos to the single-view images in the CelebA dataset. Similar to [65], we use the first 100 frames in the HDTF videos as target images and 60 images sampled from the CelebA as source images, resulting
total of 1 114,000 synthesized images. Since there is no ground truth for cross-identity reenactment, we evaluate the results based on the CSIM, AED, APD, and FID metrics. More evaluation details can be found in the supplementary.
Baselines. In terms of baselines, we compare our method against a SOTA 2D talking head synthesis method [65], two SOTA 3D head avatar animation methods [24; 29], and concurrent work [38].
4.3 Implementation Details
We implement the proposed method using the PyTorch framework [45] and train it with 8 32GB V100 GPUs. The first training stage takes 6 days. consisting of 750000 iterations. The second training stage takes 2 days with 75000 iterations. Both stages use a batch size of 8 with the Adam optimizer [30] and learning rate of 0.0001. More implementation details be found in the supplementary.
4.4 Qualitative and Quantitative Results
3D portrait reconstruction. For each testing portrait image, we reconstruct its 3D head avatar and render it from the observed view using different methods. By comparing the rendering with the input image, we assess the fidelity of 3D reconstruction of each method. Table shows the quantitative results, demonstrating that our model achieves significantly better reconstruction and fidelity scores. These results highlight the ability of our model faithfully capture details in the input images and reconstruct fideliiy 3D head avatars. Visual examples are present in the supplementary.
2 Duue to the time-consumin consuming nature of HeadNeRF, we compare our method with HeadNeRF on subset of
3000 images from the CelebA dataset.
7

---

(a) Inputs
HezdNeRFF
[b] ROME
[c] StyleHicat
(d) Ours (multipc views)
source
brastt
tanges
Figure 3: Cross-identity reenactment on CelebA [32] and HDFT [71]. The first two rows show cross-identity reenactment results on the CelebA dataset, while the last two rows demonstrate motion transfer from videos in the HDFT dataset to images in the CelebA dataset.
Table 2: Comparison on the HDTF dataset [71].
Same-Identity Reenactment
Cross: Identit Reenactment
Methods
PSNR*
SSIM
CSIM
AED]
APD! AKD]
LPIPS
L1
FID]
CSIM
AED]
APDJ.
FID]
ROME ROME|29]
20.75
0.838
0.746
0.123
0.012
2,938
0.173
0.047
31.55
0.629
0.247
0.020
43.38
OTAvatar [38]
20.12
0.806
0.619
0.162
0.017
2.933
0.198
0.053
36.63
0,514
0.282
0.028
44.86
StyleHeat
19.18
0.805
0.654
0.141
0.021
2.843
0.194
0.056
108.3
0.537
0.246
0.025
105.1
Ours
22.15
0.868
0.789
0.129
0.010
2.596
0.117
0.037
21.60
0.643
0.263
0.018
47.39
Table 3: Cross-identity reenactment between the
Cross-identity reenactment. Fig. 3, and Fig. 4 showcase the qualitative results of cross- identity reenactment on the CelebA [32] and HDFT [71] dataset. Compared to the base- lines [29; 65; 38], our method faithfully re- constructs intricate details such as hairstyles,
HDTF dataset [71] and the CelebA dataset [32].
Methods
CSIM
AED
APD]
FID
ROME [29]
0.521
0.270
0.022
76.03
StyleHeat [65]
0.461
0.270
0.038
94.28
Ours
0.551
0.274
0.017
59.48
carrings, eye glasses etc. in the input portrait images. Moreover, our method successfully synthesizes realistic appearance change corresponding the target motion. For instance, our model able to synthesize plausible teeth when the mouth transitions from closed to open (e.g., row 3 in Fig. 3), also hallucinates the occluded face region when the input image in profile view (e.g., row in Fig. 3). In contrast, the mesh- based baseline [29] can neither capture photo-realistic realistic details nor hallucinate plausible inner mouths, while the 2D talking head synthesis baseline [65] produces unrealistic warping artifacts when the input portrait is in the profile view (e.g. row 2 in Fig. 3). We provide quantitative evaluations of the cross-identity reenactment results Table 1, Table and Table 3. Our method demonstrates better fidelity and identity preservation scores, showing its strong ability in realistic portrait synthesis. It is worth noting that HDTF [71] includes images that are less sharp compared to the high-fidelity images model is trained on, which may account for the slightly lower FID score Table 2.

---

source
[o] Larget
ours
(d) linear expression
(e) w/o Aneestra
w/o appearance branch
Figure 5: Ablation studies. Details are explained in Sec. 4.5.
Table 4: Ablation studies. Blue text highlights the inferior performance the variants. (Sec. 4.5)
3D Portrait Reconstruction
Cross-Identity Recanct
Methods
L1l
LPIPS)
PSNR
SSIM
FID.
CSIMt
AED
APDJ
FID,
perspective
0.061
0.239
19.53
0.370
0.243
0.015
w/o neutral constraint
0.027
0.124
25.65
0.854
13.56
0.593
0.315
0.018
22.92
linear expression
0.031
0.134
25.08
0.841
14.62
0.443
0.217
0.016
26.18
Ours
0.030
0.116
10.47
0.276
0.017
17.36
Same-identity reenactment. Ta- ble shows the quantitative results of same-identity reenactment on
target
(c) ROME
(4) StyleHeat
(1) Ours
HDTF [71]. Our method generalizes well to HDTF and achieves better existing SOTA methods [29; 38; 65]. The qualitative results of sume-identity reenactment can be found in the supplementary.
metrics compared
Figure 4: Cross-identity reenactment on HDFT [71].
Efficiency. Since HeadNeRF [24]
and OTAvatar [38] require latent code optimization for unseen identities, process of recon-
structing and animating an avatar takes them 53.0s and 19.4s, respectively. Meanwhile, ROME [29]
and our method only needs an efficient forward pass of the network for unseen identities, taking 1 1.2s and 0.6s respectively. Overall, our method strikes the best balance in terms of speed and quality.
4.5 Ablation Studies
We conduct experiments to validate the effectiveness of the neutral expression constraint (Sec. 3.1), the contribution of the appearance branch (Sec. 3.2), and the design of the expression branch (Sec. 3.3).
Neutral expression constraint. In model, we aim wipe the expression in the source image by enforcing the coarse reconstruction from the canonical branch to have neutral expression. This ensures that the expression branch always animates fully "neutralized" expressionless face from the canonical branch. Without this, the expression branch fails to correctly modify the coarse reconstruction into the target expression, as shown Fig. 5(e) and Table (i.e., worse AED score).
Appearance branch. The appearance branch is the key to reconstructing intricate facial details of the input portrait image. Without this branch, the model struggles capture photo-realistic details, resulting in considerably lower reconstruction and fidelity metrics, as shown in Table and Fig. 5(f).
Alternative expression branch design. Instead of using the frontal view rendering from the 3DMM to provide target expression information to the expression branch (see Sec. 3.3), an alternative way to use the 3DMM expression coefficients to linearly combine set of learnable expression bases. However, this design performs sub-optimally as it overlooks the individual local deformation details caused by expression changes, introducing artifacts (e.g., mouth not fully closed) shown in Fig. 5(d) and lower FID in Table 4. We provide more results and ablations in the supplementary.
Conclusions
In this paper, we propose framework for one-shot 3D human avatar reconstruction and animation from a single-view image. Our method excels at capturing photo-realistic details in the input portrait image, while simultaneously generalizing to unseen images without the need for test-time optimization. Through comprehensive experiments and evaluations on validation datasets, we demonstrate that the proposed approach achieves favorable performance against sfate-of-the-ar baselines head avatar reconstruction and animation.

---

References
[1] A 3D Face Model for Pose and Illumination Invariant Face Recognition, 2009. 2
[2] ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu. Rignerf: Fully
controllable neural 3d portraits. In CVPR, 2022. 2,3
[3] Yunpeng Bai, Yanbo Fan, Xuan Wang, Yong 7hang. Jingxiang Sun, Chun Yuan, and Ying Shan. High-fidelity facial avatar reconstruction from monocular video with generative priors. arXiv preprint arXiv:22/1 1,5064, 2022. 3
[4] Ziqian Bai, Feitong Tan, Zeng Huang, Kripasindhu Sarkar, Danhang Tang, Di Qiu, Abhimitra Mcka, Ruofei Du, Mingsong Dou, Scrgio Orts-Escolano, et al. Learning personalized high quality volumetric
head avatars from monocular rgb vidcos. arXiv preprint arXiv:2304.01436. 2023. 2, 3
[5] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. 2CCV. 2. [6] Ananta R Bhattarai, Matthias Nießner, and Artem Sevastopolsky. Triplanenet: An encoder for eg3d inversion. arXiv preprint arXiv: 1..03.13499 13497, 2023.
[7] Volker Blanz and Thomas Vetter. A morphable model for the symthesis of 3d faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 1999. 2
[8] Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the & 3d face alignment prohlem? (and a dataset of 230,000 3d facial landmarks). In ICCV, 2017.7
[9] Eric R. Chan, Connor Z. Lin. Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay. Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient scometry-aware 3D generative adversarial networks. In CVPR, 2022. 2, 3, 4, 6.
[10] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial Radiance Fields.
In ECCV, 2022. 2,
[11] Radek Danecek. Michael J. Black, and Timo Bolkart. EMOCA: Emotion driven monocular face capture
and animation. CVPR. 2022. 1, 2
[12] Jiankang Deng. Jia Guo, Xue Niannan, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In CVPR, 2019.
[13] Yu Deng, Baoyuan Wang, and Heung- Yeung Shum. Learning detailed radiance manifolds for and 3d-consistent portrait synthesis from monocular image. arXiv preprint arXiv:2211. 1,,992211.13001,, 2022. 3, [14] Deng, Jiaolong Yang. Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruc- tion with weakly-supervise learning: From single image to image set. In CVPRW, 2019. 1, 3, 4, 5,
[15] Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Alcksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits:
One-shot megapixel neural head avatars. 2022. 1,2
[16] Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al. 3d morphable face models past,
present, and future. ACM Transactions on Graphics (TOG), 2020. 2
[17] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. .earning an animatable detailed 3D face model from mn-the-wild images. SIGGRAPH. 2021. 1,2
[18] Anna Frühstück. Nikolaos Sarafianos, Yuaniu Xu, Peter Wonka, and Tony Tung. VIVE3D: Viewpoint-
independent video editing using 3D-aware GANs. In CVPR, 2023. 3
[19] Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In CVPR, 2021. 2,3
[20] Chen Gao, Yichang Shih, Wei-Sheng Lai, Chia- Liang, and Jia-Bin Huang. Portrait neural radiance fields from a single image. arXiv preprint X00:: 2.50035, 2020. 2,
[21] Xuan Gao, Chenglai Zhong, Jun Xiang, Yang Hong, Yudong Guo, and Juyong Zhang. Reconstructing personalized semantic facial nerf models from monocular video. SIGGRAPH Asia, 2022. 2, [22] lan Goodfellow, Jean Pouget-Abaade
Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron
Courville, and Yoshua Bengio. In NewrlPS, 2014. 2, 6
[23] Yudong Guo, Keyu Chen, Sen Liang, Ying-Jin Liu. Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In ICCV, 2021. 2.
[24] Yang Hong, Bo Peng, Haiyao Xiao, Ligang Liu, and Juyong Zhang. Headnerf: A real-time nerf-based parametric head model. In CVPR, 2022. 3, 7.9
[25] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. ECCV, 2016.
[26] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[27] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020. 1, 6
[28]
Zhanghan Ke, Jiayu Sun, Kaican Lii, Qiong Yan, and Rynson W.H. Lau. Modnet: Real-time trimap-free
portrait matting via objective decomposition. In AAAI, 2022. 7
[29] Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Egor Zakharov. Realistic one-shot bessh
head avatars. In ECCV, 2022. 1. 3, 6, 7, 8,
[30] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv: 13Xiv:4122.990 1412.6980, 2014. 7
[31] Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, and Matthias Nießner. Nersemble:
Miewi-view radiance field reconstruction of human heads, 2023. 3
10

---

[32]
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial
image manipulation. In CVPR, 2020. 6, 7.
[33]
Yeonkycong Lcc, Tacho Choi, Hyunsung Go. Hyunjoon Lec, Sunghyun Cho, and Junho Kim. Exp-gan gan:
3d-aware facial image generation with expression control. In ACCV, 2022. 3, 6
[34]
Biwen Lei, Jianqiang Ren, Mengyang Feng, Miaomiao Cui, and Xuansong Xie. A hierarchical representa-
tion network for accurate and detailed face reconstruction from in-the- wild images. 2023. 2
[35] Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, et al. Learning formation of physically-base face attributes. In
CVPR, 2020. 2
[36] Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4D scans, ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 2017. 2, 3 [37] Steven R I Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one. 6
[38] Zhiyuan Ma, Xiangyu Zhu, Guojun Qi. Zhen Lei, and Lei Zhang. Otavatar: One-shot talking face avatar
with controllable tri-plane rendering. arXiv preprint ariivi2303 14662, 2023. 1,3, 6, 8, 9
[39] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 1,2,
[40] Maryam Sadat Mirzaci, Kourosh Meshgi, Etienne Frigo, and Toyoaki Nishida. Animgan: generative adversarial network for character animation. ICIP, 2020.
spatiotemporally-condiiii
[41] Thomas Müllrr, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 2022. 2,
[42] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Bruale Nerfies: Deformable neural radiance fields. ICCV, 2021. 2,
[43] Keunhong Park. Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman. Ricardo Bruulla, and Steven M. Seitz. Hypemerf: A higher-dimensiona.
representation for
topologically varying neural radiance fields. ACM Trans. Graph., 2021. 2,3
[44] Frederic Ira Parke. A parametric model for human faces. The University of Utah, 1974.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito Zeming Lin, Alhan Desmaison, Luca Antiga, and Adam Lerer. Automatic di differentiation in pytorch. 2017. 7 [46] Yurui Ren, Ge Li. Yuanqi Chen, Thomas H Li, and Shan Liu. Pirenderer: Controllable portrait image
[45]
generation via semantic neural rendering. In JCCV, 2021. 1,2 [47] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Trans. Graph. 2021. 3
[48] Aliaksandr Siarohin. Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion
model for image animation. NeurlPS, 2019. 1.2
[49] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni-
tion. arXiv preprint arXiv: 1409. 1556, 2014. 4
[50] Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, and Yebin Liu. Next3d; Generative neural texture rasterization for 3d-aware head avatars. arXiv preprint arXiv:2211.1 1208, 2022. 3.6 6
[51] Keqiang Sun, Shangzhe Wu, Zhanyang Huang, Ning Zhang, Quan Wang, and HongSheng Li. Controllable 3d face synthesis with conditional generative occupancy fields. arXiv preprint (rXiv:2206008361 2022. 3,
[52] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. arXiv preprint 2x3v:2303.14184, 2023.
[53] Junshu Tang, Bo Zhang, Binxin Yang, Ting Zhang, Dong Chen, Lizhuang Ma, and Fang Wen. Explicitly
controllable 3d 3wwae portrait generation. arXiv preprint arXiv:2209.00 2204445444. 2022. 3
[54] Kartik Tcotia, Mallikarjun B R, Xingang Pan, Hyeongwoo Kim, Pablo Garrido, Mohamed Elgharib, and Christian Theobalt. Hq3davatar: High quality controllable 3d head avatar. 2023. 2,
[55] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene
from monocular video. In ICCV, 2021. 2,3
[56] Alex Trevithick, Matthew Chan, Michael Stengel, Eric R Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorth,, and Koki Nagano. Real-time radiance fields for single-image portrait view synthesis. arXiv preprint arXiv:2305.02. 230.0330,, 2023. 3
[57] Ting-Ch Wang, Arun Mallya, and Ming- Yu Liu. One-shot fire-view neural talking-head synthesis for
video conferencing. In CVPR, 2021. 1, 2
[58]
Xintao Wang. Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with
generative facial prior. In CVPR, 2021. 1.
[59] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. NewrIPS, 2021. 4 [60] Jiaxin Xie. Hao Ouyang, Jingtan Piao, Chenyang Lei, and Qifeng Chen. High-fidelity 3d gan inversion by pseudo- maaii-mlliiiiiiiii optimization. arXiv preprint 2rXiv:221..1566. 2022. 3

---

[61]
Eric Zhongcong Xu, Jianfeng Zhang, Junhao Liew, Wenqing Zhang, Song Bai. Jiashi Feng. and Mike Zheng
Shou. Pv3d: A 3d generative model for portrait video generation. In ICLR, 2023. 3
[62]
Hongyi Xu, Guoxian Song, Zihang Jiang, Jianfeng Zhang. Yichun Shi, Jing Liu, Wanchun Ma. Jiashi Feng, and Linjie Luo. Omniavatar: Geometry-guider controllable 3d head synthesis. arXiv preprint
arXiv:2303. 155339, 2023.
[63]
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann.
Point-nerf: Point-based based neural radiance fields. arXiv preprint arXiv:2201.0888 08885,, 2022.
[64]
Yiran Xu, Zhixin Shu, Cameron Smith, Jia- Bin Huang, and Scoung Wug Oh. In-n-out: Face vidco inversion and editing with volumetric decomposition. arXiv preprint arXiv:2302.0433
2023. 3
[65]
Fei Yin, Yong Zhang. Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang. Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution editable talking face generation via
pre- pre-trained stylegan. In ECCV. 2022. 2. 6, 7. 8.9
Ziyang Yuan, Yiming Zhu, Yu Li, Hongyu Liu, and Chun Yuan. Make encoder great again in 3d gan inversion through geometry and occlusion-awwre encoding, arXiv preprint arXiv:2303. 12326, 2023, 3, [67] Egor Zakharov, Alcksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky, Fast bi-layer neural
[66]
synthesis one-shot realistic head avatars. In ECCV, 2020. 1,2
[68] Jingbo Zhang. Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Fdnerf: Few-shot dynamic neural radiance fields for face reconstruction and expression editing, In SIGGRAPH Asia, 2022.
[69] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018. 4
[70] Wenxuan Zhang, Xiaodong Cun, Xuan Wang. Yong Zhang. Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation, arXiv preprint arxiv:22 121994 2022. 1.2
[71] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with high-resolution audio-visual dataset. In CVPR, 2021. 6, 7, 8, 9
[72] Yufeng Zheng. Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, and Otmar
Hilliges. 1M Avatar: Implicit morphable head avatars from vidcos. In CVPR, 2022. 2,
[73] Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J. Black, and Otmar Hilliges. Pointavatar:
Deformable point-based head avatars from videos. In CVPR, 2023.
[74] Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang. Siwei Tang. Li Zhang, Ziwei Liu, and Chen Change Loy.
CelebV-HQ: A large-scale video facial attributes dataset. In ECCV, 2022. 6
[75] Peiye Zhuang. Liqian Ma, Sanmi Koyejo, and Alexander Schwing. Controllable radiance fields for dynamic
face synthesis. In 3DV, 2022. 3
[76] Yiyu Zhuang, Hao Zhu, Xusen Sun, and Xun Cao. Mofanerf: Morphable facial neural radiance field. In ECCV, 2022. 3,7
12


Reviews:
[{"review_id": "wdpsE3hFyz", "reviewer": "Reviewer_ufhJ", "summary": "1. This work proposed a new framework for generalizable head animation in one-shot setting.\n2. The results are better than several baselines.", "questions": "None.", "limitations": "No. Limitations are not discussed in the main paper.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The target problem is important.\n2. The method is reasonable.", "weaknesses": "1. Task definition. 1) The practical usage situation need to be clarified? The task is like a 3D version of facial animation. However, the view points changing range is limited and the missing part is not painted. Thus, what is the actual application of this task? Why we should make facial animation 3D-aware when the view point is somewhat limited. 2) The title is miss leading. It looks like a work stressed generalizable ability. However, it is in generalizable ability and 3D-aware. In 2D, there are already a lot work that is generalizable.\n\n2. Comparison. 1) The compared baselines are insufficient. The work in 2D (such as [1,2,3] ) is still needed to be compared, even only in the unchanged view-points as driven videos'. 2) The comparison on CelebV-HQ is necessary. Recently, only HDTF is used in evaluation. However, HDTF is much simpler than CelebV-HQ. The results on a more challenging dataset CelebV-HQ can be much more convincing, both in the comparison on main paper and video.\n\n3. Limitations. Limitations should be discussed in the main paper.\n\n[1] First order motion model for image animation.\n\n[2] Thin-plate spline motion model for image animation.\n\n[3] FNeVR: Neural volume rendering for face animation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Task definition and 2D baselines.**\n- Please see A1 in the Global Response above and Fig.1 in the rebuttal PDF. We emphasize that compared to the 2D baselines, our method learns a strong 3D geometric prior of human faces and hallucinates more realistic missing parts, especially when the source portrait is captured from an extreme profile view.\n- We will change the title to “Generalizable One-shot 3D Neural Head Avatar” in the next revision.\n\n**Evaluation dataset.**\n- We have discussed the training and evaluation datasets in Section 4.1 of the main paper. Including the baselines added in this rebuttal, we have compared our method to 13 baseline methods in total, each of which is trained with different training datasets. It is infeasible for us to re-train all of them on the same training datasets yet guarantee to achieve their best performance. Thus we follow prior work [1] and carry out evaluation on the *CelebA-HQ* dataset and the testing split of the HDTF dataset. We believe this is a fair comparison since none of the baseline methods, nor our method, has seen these two evaluation datasets during training. Meanwhile, CelebV-HQ is one of our training datasets. Thus, it is not fair to other baselines if we run evaluation on it. We also emphasize that *CelebA-HQ* is a challenging dataset, which includes realistic portrait images with high fidelity details.\n\n[1] Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. ECCV, 2022.\n\n**Limitations.**\n- We have discussed the limitations and the social impact of our work in detail in the supplementary and will move them to the main paper in the next revision."}, {"review_id": "7CAUtlK2QD", "reviewer": "Reviewer_FcT2", "summary": "This paper presents a method for generating Neural Head Avatars given a single image of a subject. More specifically, the input is a source image of a person as well as a target image specifying the expression, and the goal is to generate a rendering of the person in the source image with the expression of the target image. To do so, the method predicts 3 sets of tri-planes modelling the coarse geometry and appearance of the source image, and the expression of the second image. The geometry and appearance are modeled in a canonicalized 3D pose to encourage robustness to different poses and varying viewpoints. After training, the method can be applied to unseen images and generate animatable avatars without requiring any expensive test-time optimization. Overall the experiments validate the effectiveness of the approach.", "questions": "1. I was wondering how accurate is the 3DMM prediction stage, and how much this affects the overall quality of the reconstructions.", "limitations": "The paper fails to discuss the potential societal impacts of this work. One obvious misuse of the technology could be in DeepFakes. I believe that the authors should include a more detailed overview of the potential misuses and what safeguards can be used.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The methodology of the paper is technically sound. The components of the method are sufficiently motivated and their usefulness is assessed in the ablation study.\n2. The paper has very good experimental results. Quantitatively, it performs favorably compared to the state-of-the-art in the CelebA and HDTF datasets. The qualitative results are also very good.\n3. The paper is well-written and easy to follow. Most components are adequately described in the main text and the figures are relatively easy to parse (with the exception of some parts of Figure 1. where the flow of information is not immediately clear). ", "weaknesses": "The paper does not have any particularly important weak points. Occasionally there are some artifacts in the reconstruction for parts of the face that are not visible in the source image (e.g. Row 2 of Figure 3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**3DMM accuracy.**\n- Since our expression branch solely relies on the 3DMM renderings to provide target expression information, it is indeed affected by the 3DMM's accuracy. When the 3DMM model fails to predict the target expression correctly from the target image, our model replicates the inaccurate expression in the 3D reconstruction as well. We thank the reviewer for pointing out this limitation and we will discuss it in the next revision.\n\n**Social impact.**\n- We have discussed the limitations and the social impact of our work  in details in the supplementary and will move them to the main paper in the next revision.\n"}, {"review_id": "P5sx6hztm5", "reviewer": "Reviewer_8oNf", "summary": "The paper proposes a novel approach for building single-shot animatable head avatars. This is achieved through three branches: the canonical reconstruction branch, the detailed appearance branch, and the expression branch, all of which are represented as tri-plane nerfs. The three tri-planes are added together to form the final 3D representation. The paper also features a super-resolution module, which not only achieves much better photorealism than existing methods but also nicely hallucinates teeth while retaining multi-view consistency.\n\n", "questions": "Please see the first two points of the weaknesses section. \n", "limitations": "Yes, the author adequately discussed limitation and negative societal impact in the supplementary material. \n\n\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The usage of separate tri-planes for canonical, appearance details, and expressions is a novel approach (although there are some weaknesses associated with this design choice, as discussed in the following sections).\nAlong with the super-resolution block, the method generates high-quality images and surpasses previous works in various image quality and diversity metrics. The proposed expression branch can successfully hallucinate teeth during deformation. Additionally, multi-view consistency is well-preserved, as demonstrated in the supplementary video. Overall, the approach is thoroughly evaluated and compared with state-of-the-art (SOTA) baselines.\n\n", "weaknesses": "Weakness of the result:\nThe method has difficulty with jaw opening. In the supplementary video, when the person opens the mouth, only the lips move and not the jaw. I suspect that this is because the additive expression branch is not powerful enough to model such big deformation. What’s the author’s explanation of this?\n\nAppearance branch:\nUsing the depth of the source image, the paper proposes to back-project the source image feature to 3D space in the appearance branch. It seems to me that this branch not only adds canonical appearance details but also would entangle the source expressions into the 3D reconstruction. Since the expression branch does not take into account the source expression parameters, how does the method remove the source expression from the 3D reconstruction? Does the network learn to ignore expression-dependent details in the appearance branch through neural network magic? How does R_v(T_c + T_p) look like in Figure 2?\n\nInvalid ablation on the expression branch:\nThe paper shows that the proposed expression branch performs better compared to a ‘linear expression’ ablation. However, this ablation baseline does not make sense to me because the 3DMM expressions are linear blendshapes, i.e. the warping fields are linearly combined, not the 3D volumes. Therefore, expressions cannot be expressed as the weighted average of expression triplanes. This ablation setup is not valid in my opinion. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness of the result.**\n- Please see A2 in the Global Response above, Fig.4 in the PDF and the video we sent to AC. The \"jaw opening\" issue is caused by the long-tail expression distribution of the training dataset rather than using the additive expression branch. With training data containing a more balanced distribution of large facial expressions, our method naturally deforms the jaw when the mouth is wide open.\n\n**Appearance branch.**\n- Please see A3 in the Global Response above, the visualization of $R_v(T_c + T_p)$ can be found in Fig.2 (a) in the rebuttal PDF. In the submission, we remove the source expression by masking out the expression-dependent regions (i.e., eyes and mouth) before feeding the source image into the appearance branch. In A3 in the Global Response above, we further show that learning a model that implicitly ignores the source expression is feasible and shows promising preliminary results.\n\n**Invalid ablation on the expression branch.**\n- We will remove this ablation study in the next revision."}, {"review_id": "fCKaf92HRM", "reviewer": "Reviewer_PU1D", "summary": "The method aims to build a generalizable model to create an animatable 3D human head representation from a single-view portrait source image. The resulting representation can be used to reenact the source image with target images with different subjects and expressions.\n\nThe key idea is to use three tri-planar representations (T_c, T_p, and T_e) with the underlying 3DMM from [14].\n\nT_c is a canonical tri-plane representing the neutral source face. Since the source face can be in arbitrary expressions, this branch is responsible to undo the expression. An encoder pretrained with SegFormer [59] maps the input source image to a tri-plane with 32 feature channels. To enforce T_c to be in the neutral face, this branch relies on [14] to extract the identity and expression coefficients. The neutral expression obtained by zeroing out the expression coefficients\n\nT_p is responsible for capturing detailed appearance. Neural point cloud representation is constructed from the depth image of T_c viewed from the camera of the source image, with features associated with the source image, again obtained from an encoder pretrained with SegFormer [59]. This neural point cloud is finally rasterized to the appearance tri-plane T_p.\n\nT_e represents a tri-plane based on the 3DMM of the source identity and the target expression extracted with [14]. The frontal view rendering of this identity and expression is passed to an encoder pretrained with SegFormer [59] to obtain an expression tri-plane T_e.\n\nThe image generated from the sum of three tri-planes goes through a super-resolution with a pre-trained GFPGAN to obtain the final image.\n\nThe method is compared against ROME [29] and StyleHeat [65] in the main paper. It is also compared against a recent work Next3D in the supplementary.", "questions": "If authors have more thoughts on why the visual artifact with the jaw open and extreme expressions happens, I would like to hear them. Other than the naive composition of the tri-planes, I suspect the underlying 3DMM may not be sufficient to capture realistic deformations.", "limitations": "As mentioned in the weakness, the method exhibits clear artifacts with large facial deformations. This should be clarified.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The use of the different tri-planar representations, each making use of the information from the 3DMM prediction, is interesting.\n\nA setup only requiring a feedforward pass without test-time optimization is simple and practical.", "weaknesses": "A naive addition of three tri-plane representations may not make sense. T_c is representing the neutral face while T_p is obtained from the source image with an expression. A corresponding position on a face will have the spatial coordinates in T_c and T_p, making this addition insensible. Perhaps consider warping the tri-planes before adding them?\n\nThe first sentence in 3.2 says the appearance branch is to reconstruct intricate facial details. This may be somewhat misleading. What the appearance branch is really responsible for is capturing the details of the non-facial regions. This is clear from Fig. 2 (e) showing a better reconstruction of the hat, hair, and necklace. Fig. 5 (f) is also highlighting the earring.\n\nGiven the above, the proposed setup would have made more sense if the appearance branch masked out the facial region, in contrast to the canonical branch masking out the non-facial region when taking the loss.\n\nPerhaps related to this, the video shows visual artifacts. The result looks like the facial parts are animated textures on a non-deforming geometry. This is clear when the target subject opens the jaw at 0:20 and 1:04 but the resulting face opens the mouth without moving the jaw. This is a sign that the neutral face geometry is having too much influence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Appearance branch.**\n- Please see A3 in the Global Response above. As discussed in the supplementary (Line 105 - 108), to prevent the source expression from leaking into the final animation results, the appearance branch masks out the facial region in the input source image and only captures the non-facial details, while the expression reconstruction solely coming from the expression branch. We will clarify this more in the next revision.\n\n**Performance and artifacts in rare expressions.**\n- The proposed method efficiently generalizes a single network to unseen identities. while demonstrating competitive 3D head avatar animation performance quantitatively and qualitatively against multiple state-of-the-art baselines. As shown in Sec.4 in the main paper and Sec.3 in the supplementary, our model can produce realistic animation results for most common target expressions.\n- For the \"jaw opening artifacts\", please see A2 in the Global Response, Fig.4 in the rebuttal PDF, and the video we sent to AC. We emphasize that the mouth-wide-open expression falls into the long-tail distribution of the training dataset, causing the artifacts. It does not demonstrate a fundamental flaw in our framework's design. With proper training data discussed in A2 above, our method can naturally deform the jaw when the mouth is wide open. We also show the extracted mesh in Fig.4 in the rebuttal PDF, demonstrating that the expression branch indeed changes not only the texture but also the underline geometry of the canonical tri-plane."}, {"review_id": "t23ahTdtHl", "reviewer": "Reviewer_k8VD", "summary": "Authors proposed a method for one-shot facial animation with high level of details saved from the input image. To handle facial deformations they disentangle expression and appearance and introduces a neural point cloud renderer for that. To train a model authors used quite popular set of loss functions and data, as well as incorporated synthetic data to improve underlying geometry. Qualitative and quantitative experiments show the advantage of architectural design and superiority over existing methods.\n", "questions": "The neural point cloud introduced in the line 182 is equivalent to dense feature grid, isn’t it? This procedure also similar to latents avatar like [1, 2], but with orthographic projection.\nCould you elaborate the reasons to include synthetic images and how does affect final quality? For instance you mentioned that CelebV-HQ is static (which is debatable) is it a help somehow?\nCould you compare expression transfer? As I see in the first video results there is a clear gap of the emotional level or naturally during the reenactment. \nWhat is a time needed for an inference?", "limitations": "The method by itself has limited geometry ability to describe human heads. Based on the presented image there is not information about upper body.\nThere is not option to render 360 degree view paths.\nLack of the realistic emotions.\nI doubt that the method can be easily used in real-time applications.\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Idea to decompose geometry and appearance in canonical face is an interesting combination of recent success in neural rendering\n- The demonstrated method outputs clearly emphasise huge number of saved details and right disentanglement for identity and expression. \n- The results contains enormous level of the details compared to the demonstrated baselines.\n- Authors carefully describe each branch contributed in the resulted quality.\n", "weaknesses": "The comparison doesn’t have a latent avatar models [1, 2]. It can improve the overall experiments section since both methods produce high-resolutional results for reenactments\n\n- [1] One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing\n- [2] Megaportraits: One-shot megapixel neural head avatars.\n\nThe introduced neural point cloud can not cover huge view changes. \nSome statements are overconfident with respect to the evaluation (mostly regarding to consistent usage SOTA).\nLack of ethical section in the main text, which is important for human avatars.\n\nSince the emotions at the video are too unrealistic, I want to emphasise that comparison with the methods above is necessary to get more fair outcomes from the paper. The method looks bonded by the pre-aligned geometry and the possibility to express human level poses.\nSome minor details:\nWriting quality can be improved.\nLack of limitation section.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comparison to latent 2D avatar methods.**\n- Please see A1 in the Global Response above. Note that we cannot compare to Megaportraits since its code is not publicly available. For \"One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing\", we used a public third-party implementation because the official code is not released (we have sent the code link of this implementation to AC since we cannot add external links in the rebuttal text).\n\n**Point clouds cannot cover huge view changes.**\n- The neural point clouds indeed only cover the visible parts in the input portrait, but the canonical branch is developed to provide cues of missing parts and guarantee a realistic full reconstruction. This is exactly our main motivation for disentangling coarse geometry and intricate details into two tri-planes. The former learns to map a portrait image in any pose to a canonical coarse reconstruction implicitly in a holistic manner, while the latter explicitly lifts the partial observation from the 2D space using camera pose and rendered depth. Through training, the two branches learn to be compatible with each other and produce reasonable full reconstructions for inputs captured from various views. As shown in Fig.1 (right) in the rebuttal PDF, even for a source image captured from an extreme profile view, our method produces reasonable reconstruction, while 2D baselines present unrealistic warping artifacts. We do think that further improvement is possible for the neural point cloud part. For instance, one way is to inpaint the neural point cloud by utilizing the symmetry prior of human portraits, we leave this exploration in future works.\n\n**Synthetic images.**\n- Why do we need synthetic data? We first analyze the variation between source and target poses in the CelebV-HQ and the synthetic datasets. We randomly sample 1000 training pairs and compute the pose distance between the source and target image in each pair. The average pose difference in CelebV-HQ and the synthetic dataset is 0.0482 and 0.0985, respectively. This shows that sampled pairs in CelebV-HQ have less pose variation. Instead, the source and target images in the synthetic data have larger pose variation, which enforces the model to hallucinate realistic missing parts during training. We further carry out an ablation study that learns a model without using synthetic data. As shown in the last column in Fig.4 in the rebuttal PDF,  the model learned without synthetic data shows artifacts in the occluded region (e.g., the left side of the nose).\n- What alternative generative models could be used to produce synthetic data and how does it affect our model? We kindly point to A2 in the Global Response above, where we show that by replacing EG3D with Next3D [2] in the training data synthesis process, our model can be further improved. \n- We would like to emphasize that utilizing pre-trained generative models for data synthesis demands minimal effort. In our case, the synthesis of over 50,000 training images using either EG3D or Next3D takes just a few hours.\n\n[2] Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars\n\n**Inference time.**\n- We have discussed efficiency in Line 333 - 337 in the main paper. We emphasize that our method is efficient since it is generalizable and does not require any person-specific optimization or test-time fine-tuning. Real-time is not a primary focus in this paper. However, we do identify possible acceleration methods such as replacing SegFormer encoders with more light-weight networks, reducing rendering resolution, or using mixed precision inference.\n\n**Upper body and 360 degree view paths.**\n- Our method outperforms existing methods by capturing high fidelity details in regions beyond the face area (e.g., hair, earrings). Upper body human capture is related to human body deformation and especially challenging in the generalizable setting. Even in works that overfit a model to a specific identity [3], reconstructing the upper body is non-trivial and requires a dedicated neural radiance field. We leave upper body reconstruction for future research. Similarly, 360 degree head reconstruction requires advanced representations than the native tri-plane and special training data, as discussed in [4]. Though we believe that our method can generalize to full heads using the new techniques and data in [4], it is not the focus of this paper and is left out for future research.\n\n[3] AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis. ICCV 2021.\n\n[4] PanoHead: Geometry-Aware 3D Full-Head Synthesis in 360deg. CVPR. 2023.\n\n**Ethical and limitation section.** \n- We have discussed the limitations and the social impact, in detail, in the supplementary document and will move them to the main paper in the next revision.\n\n**Overconfident statements.**\n- We will remove SOTAs.\n\n**Limited geometry ability.**\n- Please see A2 in the Global Response, Fig.4 in the rebuttal PDF, and the video we sent to AC.  We show the extracted mesh in Fig.4 in the rebuttal PDF, demonstrating that the expression branch can deform the underline geometry of the canonical tri-plane to match extreme expressions (e.g., wide open mouth) so long as we use balanced training data."}]

Author rebuttal:
### A1. Comparison to 2D baselines (**ufhJ**, **k8VD**)

**Motivation.**

We discuss the benefits of 3D avatars towards talking head synthesis in Lines 35 - 37 and Lines 317 - 319 of the main paper. Other important reasons to study 3D avatars are:
- Human faces are inherently 3D, thus it is physically accurate to model faces in 3D. As shown in Fig.1 in the PDF, compared to 2D baselines, our method has fewer warping artifacts, produces consistent geometry across different views, and is robust to profile view inputs.
- For practical usage, mentioned by **ufhJ**, a 3D avatar is essential in immersive AR / VR applications so that a person can be rendered from novel views to convey eye contact. Meanwhile, 2D methods only change head pose but do not address novel view synthesis easily. Furthermore, traditional pipelines (e.g., gaming) require 3D assets as inputs, which can be extracted from our method.

**Experiments.**

To empirically verify the advantages of our method compared to 2D baselines, we carry out cross-identity reenactment on CelebA-HQ and show results below:

| Method  | CSIM &uarr; | AED &darr;| APD &darr;| FID &darr;| Year|
| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |
| [1] FOMM  | 0.528  | 0.284 | 0.034 | 37.31 | NeurIPS 2019|
| [2] FaceVid2Vid  | 0.638  | 0.284 | 0.037 | 24.66 | CVPR 2021|
| [3] Thin-Plane | 0.556| 0.272| 0.029| 34.75| CVPR 2022|
| [4] DaGAN | 0.502| 0.276| 0.032| 36.78| CVPR 2022|
| [5] FNeVR | 0.520| 0.283| 0.031| 33.18| NeurIPS 202
| [6] LIA | 0.607| 0.274| 0.037| 34.33| ICLR 2022|
| [7] DPE | 0.635| 0.285| 0.048| 43.38| CVPR 2023|
| [8] MCNET |0.491| 0.262 | 0.030 | 38.89| ICCV 2023|
| Ours | **0.649**| 0.269| **0.018**| **18.68**| |
| Ours* (A3 below) | 0.716| 0.262| 0.019|20.70| |

[1] First order motion model for image animation.

[2] One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing.

[3] Thin-plate spline motion model for image animation.

[4] Depth-Aware Generative Adversarial Network for Talking Head Video Generation.

[5] FNeVR: Neural volume rendering for face animation.

[6] Latent Image Animator: Learning to Animate Images via Latent Space Navigation.

[7] DPE: Disentanglement of Pose and Expression for General Video Portrait Editing.

[8] Conditioned Memory Compensation Network for Talking Head video Generation.

Overall, our method has better performance. Qualitative results are shown in Fig.1 in the PDF.  In the left example, our method preserves the geometry of the input, while the baselines all "squeeze" the head. In the right example, our method hallucinates realistic missing parts with high fidelity (e.g., wrinkles around the eyes) when the source image is captured from a profile view, while all baselines show warping artifacts.

---
### A2. Jaw opening issue (**8oNf**, **PU1D**)

This issue is caused by the long-tail expression distribution of the training data, rather than being a limitation of our method, (i.e., “mouth wide open” is rare in the training dataset). To verify, we replace EG3D with a deformable 3D GAN [1] for data synthesis, which has two advantages: a) each training pair includes different expressions that encourage the model to produce more accurate deformation. b) the data synthesized by [1] includes larger and more balanced facial deformation. We re-train our method using this dataset and show qualitative results in Fig.4 in the PDF. This model realistically deforms the jaw when the mouth is open, validating our hypothesis. Please also see the video sent to AC.

[1] Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars.

---
### A3. Learning warping in the appearance branch. (**FcT2**, **PU1D**)

**How does the appearance branch ignore source expression?**

- As discussed in the supplementary (Line 105 - 108), we mask out expression-dependent regions (i.e., eyes and mouth) in the source image before it is input into the appearance branch. This way, expression information solely comes from the expression branch.
- As suggested by **8oNf**, we visualize the rendering of $R_v(T_c + T_p)$ in Fig.2 (a) in the PDF, which is a neutral face with intricate details. This shows that $T_p$ adds details while preserving the expression in $T_c$.

**Can we learn warping in the appearance branch?**

- Although the “masking” strategy effectively removes the source expression in the appearance branch, the warping idea suggested by **PU1D** and **8oNf** is inspiring. Interestingly, we found that simply using the appearance encoder to implicitly warp and fuse the appearance and target expressions works best (see Fig.3 in the PDF) as opposed to explicit warping. Specifically, we keep the canonical branch unchanged and merge the appearance and expression branches. This merged branch inputs the concatenation of the source image, canonical rendering, 3DMM rendering with source and target expressions respectively and the source image's mouth mask to an encoder that predicts 2D features. Then it uses the “Lifting” and “Rasterization” operation (Sec. 3.2 in the paper) to produce a tri-plane $T_{ae}$ that encodes both the appearance and the target expression. The intuition is to make the encoder $E_{ae}$ aware of both expressions (from the 3DMM renderings and the mouth mask) and the appearance (from the source image) simultaneously, such that it implicitly learns to transfer details from the source image to proper locations in  $T_{ae}$. We show results in Fig.2 (b) in the PDF and evaluation in the above table (i.e., Ours*). This model has a higher CSIM, because without masking out eyes and mouth in the appearance branch, the model preserves eye and lip color well.  *We emphasize that the “masking” strategy in the paper is sufficient to remove the source expression, but we thank the reviewers for the inspiring warping idea. The preliminary experiment above proves that this idea is feasible and introduces new directions for future work.*

Meta-review: This paper received mixed scores. All reviewers posed some issues and the authors also provided sufficient explanations and additional experimental results. Finally, most of the reviewers are in positive side. Evenly the remaining two reviewers did not change the opinion, they also agree on the acceptance. The AC finally vote to accept, considering 1) the three-branch tri-plane design is novel; 2) the results are SOTA; 3) the results and explanations are sufficient. The authors still need to include all additional results and explanations in the rebuttal to the final revision.
Decision: Accept (poster)