GPAvatar: Generalizable and Precise Head Avatar from Image(s)
Authors: Xuangeng Chu
Yu Li
Ailing Zeng
Tianyu Yang
Lijian Lin
Yunfei Liu
Tatsuya Harada
Venue: ICLR 2024 poster (2024)
Keywords: NeRF
Head Avatar
Dynamic NeRF
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Head avatar reconstruction, crucial for applications in virtual reality, online meetings, gaming, and film industries, has garnered substantial attention within the computer vision community. The fundamental objective of this field is to faithfully recreate the head avatar and precisely control expressions and postures. Existing methods, categorized into 2D-based warping, mesh-based, and neural rendering approaches, present challenges in maintaining multi-view consistency, incorporating non-facial information, and generalizing to new identities. In this paper, we propose a framework named GPAvatar that reconstructs 3D head avatars from one or several images in a single forward pass. The key idea of this work is to introduce a dynamic point-based expression field driven by a point cloud to precisely and effectively capture expressions. Furthermore, we use a Multi Tri-planes Attention (MTA) fusion module in tri-planes canonical field to leverage information from multiple input images. The proposed method achieves faithful identity reconstruction, precise expression control, and multi-view consistency, demonstrating promising results for free-viewpoint rendering and novel view synthesis.

Published as a conference paper at ICLR 2024
GPAVATAR: GENERALIZABLE AND PRECISE HEAD AVATAR FROM IMAGE(S)
Xuangeng Chu Yu Li
Ailing Zeng"
Tianyu Yang²
Lijian Lin"
Yunfei Liu²
Tatsuya Harada 1.3
'he University of Tokyo
21nternational Digital Economy Academy (IDEA)
RIKEN AIP
{xuangeng chu, harada]@mi.t..c ....ookyo ac.jp
[liyu, zengailing, yangtianyu linlijian, liuyunfeileidea. edu, cn
Figure I:
Our GPAvatar is able to reconstruct 3D head avatars from even a single input (i.e., one- shot), with strong generalization and precise expression control. The lefimost images are the inputs, and the subsequent images depict reenactment results. Inset images display the corresponding driv- ing faces. Additionally, the first row shows three novel view results.
ABSTRACT
Head avatar reconstruction, crucial for applications in virtual reality, online meet- ings, gaming, and film industries. has garnrred substantial attention within the computer vision community. The fundamental objective of this field is to faithfully recreate the head avatar and precisely control expressions and postures. Existing methods, categorized into 2D-based warping, mesh-based, and neural rendering approaches, present challenges in maintaining multi-view consistency, incorpo- rating non-facial information, and generalizing to new identities. In this paper, we propose a framework named GPAvatar that reconstructs 3D head avatars from one several images in single forward pass. The key idea of this work is intro- duce a dynamic point-based expression field driven by point cloud to precisely and effectively capture expressions. Furthermore, we tention (MTA) fusion module in the tri-planes canonical field to leverage informa- tion from multiple input images. The proposed method achieves faithful identity reconstruction, precise expression control, and multi-view consistency, demon- strating promising results for free-viewpoint rendering and novel view synthesis. Code and demos are available at https://xg-chu https://fe-chu.gguubb.iirrpeeep htt.gggbbboopgggeeeggg/////// gpaavttar.....
Multi Tri-planes At-
INTRODUCTION
Head avatar reconstruction holds immense potential various applications, including virtual reality, online meetings, gaming, and the film industry. In recent years, this field has garnered significant at- tention within the computer vision community. The primary objective of head avatar reconstruction faithfully recreate the source head while enabling precise control over expressions and posture. This capability will facilitate the generation of desired new expressions and poses for the source portrait (Li al., 2023a; Yu al., 2023b; Li al., 2023b).
"This work was partially done during an internship at IDEA.
Corresponning Author.

---

Published as a conference paper at ICLR 2024
Some exploratory methods have partially achieved this goal and can be roughly categorized into three types: 2D-based warping methods (Yin et al., 2022), mesh-based methods (Khakhulin et al., 2022), and neural rendering methods (Sun et al., 2023; Ma et al., 2023; Li et al., 2023a; Yu et al., 2023b; Li et al., 2023b). Among these, 2lD-hased methods warp the original image to new expres- sions with warping field estimated from sparse landmarks, and then synthesize the appearance through an encoder and decoder. However, these methods struggle to maintain multi-view consis- lency when there are significant changes in head pose due to their lack of necessary 3D constraints. Furthermore, these methods are unable to effectively decouple expressions and identity from the source portrait, leading to unfaithful driving results. Mesh-based methods explicitly model the source portrait with a 3D Morphable Model (3DMM) (Blanz & Vetter, 1999; Paysan et al., 2009; Li et al., 2017; Gerig et al., 2018). By incorporating 3D information, these methods effectively address the issue of multi-view consistency. However, due to the limitations in the modeling and expressive capacity of 3DMM, the reconstructed head often lacks non-facial information such as hair, and the expressions are often unnatural. With the outstanding performance of NeRF (neural radiance field) in multi-view image synthesis, the latest methods have started to leverage NeRF for head avatar reconstruction(Xu et al., 2023; Zielonka et al., 2023; Zheng al., 2022; Sun et al., 2023; Ma al., 2023). Compared to 2D and mesh-based methods, NeRF-based methods have shown the ability to synthesize results that are 3D-consistent and include non-facial information. However, these meth- ods can't ' generalize well to new identities. Some of these methods require a large amount of portrait data for reconstruction, and some involve coneuunnn optimization processes during inference.
In this paper, we present a framework for reconstructing the source portrait in a single forward pass. Given one several unseen images, our method reconstructs an animatable implicit head avatar rep- resentation. Some examples are shown in Fig. 1. The core challenge lies in faithfully reconstructing the head avatar from single image and achieving precise control over expressions. To address this issue, we introduce point cloud-driven dynamic expression field to precisely capture expressions and use a Multi Tri-planes Attention (MTA) module in the tri-planes canonical field to leverage in- formation from multiple input images. The 3DMM point cloud-driven field provides natural and precise expression control and facilitates identity-expressin capsulate feature space that includes faithful identity information from the source portrait while modeling parts not covered by the 3DMM, such as shoulders and hair. The experiment verifies that our method generalizes well to unseen identities and enables precise expression control without test-time optimization, thereby enabling tasks such as free novel view synthesis and reenactment.
decoupling. The merged tri-planes en-
The major contributions of our work are follows:
We introduce a 3D head avatar reconstruction framework that achieves faithful reconstruc- tion in a single forward pass and generalizes well in-the-wild images.
We propose dynamic Point-based Expression Field (PEF) that allows for precise and natural cross-identity expression control.
We propose Multi Tri-planes Attention (MTA) fusion module to accept an arbitrary num- ber of input images. It enables the incorporation of more information during inference, particularly beneficial for extreme inputs like closed eyes and occlusions.
RELATED WORK
2.1 TALKING HEAD SYNTHESIS
Previous methods for head synthesis can be categorized into deformation-based, mesh-based, and NeRe-based methods. Warping- based methods(Siarohin et al., 2019; Zakharov et al., 2020: Wang et al.. 202 Yin et al., 2022; Drobyshev et al., 2022; Zhang et al., 2023) are popular among 2D generative methods. Usually, these methods apply deformation operations to the source image to drive the motion in the target image. Due to lack of clear understanding and modeling of the 3D geometry of the head avatar, these methods often produce unrealistic distortions when the poses and expressions change lot. Many subsequent works(Ren et al., 2021; Yin et al., 2022; Zhang et al., 2023) alleviated this problem by introducing 3DMM(Blanz & Vetter, 1999; Paysan et al., 2009: Li et al., 2017; Gerig et al., 2018), but this problem still exists and limits the performance of 2D methods. To completely address this problem, many 3DMM-based works(Feng et al., 2021; Danecek et al., 2022; Khakhulin et al., 2022) reconstruct animatable avatars by estimating 3DMM
2

---

Published as a conference paper at ICLR 2024
parameters from portrait images. Among them, ROME(Khakhulin et al., 2022) estimates the 3DMM parameters, the offset of mesh vertex and the texture to render the results. However, although 3DMM provides strong priors for understanding the face, it focuses only facial regions and cannot capture other detailed features such as hairstyles and accessories, and the fidelity is limited by the resolution of meshes, resulting in unnatural appearances in the reenactment images.
NeRF(Mildenhall et al., 2020) is type of implicit 3D scene representation method known for its ex- cellent performance in static scene reconstruction. Many works(Park et al., 202 la;;; Tretschk al., 2021) try to extend it from static scenes to dynamic scenes, and there are also many works al.. 2021; Zheng et al.. 2022; Xu et al., 2023; Zielonka et al., 2023; Athar al., 2023) that apply NeRF to human portrait reconstruction and animation. One of the research directions is to gen- erate controllable 3D head avatars from random noise (Sun et al., 2023; Ma et al., 2023). While these methods can produce realistic and controllable results, achieving reconstruction requires GAN inversion, which is impractical in real-time scenarios. Another research direction to utilize data from specific individuals for reconstruction (Gafni et al., 2021; Athar et al., 2022; Zheng al., 2022; Xu et al., 2023; Bai et al., 2023; Zielonka et al., 2023). While the results are impressive, they cannot learn networks for different identities and require thousands of frames of personal image data, rais- ing privacy concerns. At the same time, there are also some methods to use audio drivers to control avatar (Tang et al., 2022; Guo et al., 2021; Yu et al., 2023a), providing users with more flexible and easy-to-use driver method.
2.2 ONE-SHOT HEAD AVATARS
Sardere
20484
Ereocen VTA
JDMM
Oet FEEUR
(a) HoleNeRF
(b) GOAvser
NOFA
(d) Ours
Differences from existing state-of-thh-are methods. Existing methods may over-process expression information or use expression features, leading to expression detail loss. Our approach avoids this loss with point-based expression field, and method flexibly accepts single multi- ple images as input, enhancing information gathering through our multi-tri-planes attention module.
Figure 2:
To address these issues, some works(Trevithice et al., 2023; Hong et al., 2022; Li et al., 2023a;b; Yu et al., 2023b) have focused on reconstructing 3D avatars from arbitrary input images. Some methods can achieve static reconstruction (Trevihhiiriik et al., 2023; Hong et al., 2022), but they unable to reanimate these digital avatars. There are also methods that utilize NeRF to achieve animatable one- shot forward reconstruction of target avatars, such as GOAvatar(Li et al., 2023b), (Yu al., 2023b), and HideNeRF(Li et al., 2023a). GOAvatar (Li et al., 2023b) utilizes three sets of tri-planes to respectively represent the standard pose, image details, and expression. It also employs fine- tuned GFPGAN (Wang et al., 202 network to enhance the details of the results. NOFA (Yu et al., 2023b) utilizes the rich 3D-consistent generative prior of 3D GAN synthesize neural volumes of different faces and employs deformation fields to model facial dynamics. HideNeRF (Li et al., 2023a) utilizes a multi-resolution tri-planes representation and 3DMM-based deformation field to generate reenactment images while enhancing identity consistency during the generation process. While these methods produce impressive results, they still have some limitations in expression- driven tasks. Some of these methods either rely the rendering results of 3DMM input to control deformation fields or directly use 3DMM parameters for expression-driven tasks. In such encoding and decoding processes, subtle facial expression information may inevitably be lost.
In this paper, we utilize the FLAME (Li et al., 2017) point cloud as prior and propose a novel 3D head neural avatar framework. It not only generalizes to unseen identities but also offers precise control over expression details during reenactment and surpasses all previous works in reenactment image quality. Fig. 2 illustrates the differences between our method and existing approaches.
3

---

Published as a conference paper at ICLR 2024
PEF
Por
Parsemteers
f Ft
f
MTA
SR
Reccaing
Rouers
(cas)
Figure 3: Overview: Our method mainly comprises branches: one that captures fine-grained expressions with PEF (Sec. 3.2) and another that integrates information from multiple inputs through MTA (Sec. 3.1, 3.3). Finally, there is the rendering and super-resolution component (Sec. 3.4).
3 METHOD
In this section, we will describe our method. Our approach has the capability to faithfully reconstruct head avatars from any number of inputs and achieve precise reenactment. The overall process can be summarized as follows:
R(MTA(E(T.)) PEF((LLMEE (t))
where I; represents the input image(s), 8; is the shape parameter of the source image I,, 6r and is the desired expression and pose parameter. The canonical feature space is constructed by E(I:), and our Point-based Expression Field (PEF) is built with point cloud FLAME(s;; et, Pt) and point feature 0. If there are multiple input images, their canonical feature space will be merged by the Multi-Tri-planes Attention module (MT Finally, R is the volume rendering function that renders the reenactment image I, based camera pose Peam The overall process is illustrated in Fig. 3. In the following. we will describe the canonical encoder in Sec. 3.1, explain how PEF controls expressions in Sec. 3.2, introduce how fuse multiple inputs through MTA in Sec 3.3, discuss the rendering and super-resolution process in Sec. 3.4, and finally, we describe the training targets in Sec. 3.5.
3.1 CANONICAL FEATURE ENCODER
Due to the fact that the tri-planes representation has strong 3D geometric priors and strikes good balance between synthesis quality and speed, we employ the tri-planes representation as our standard feature space. Specifically, inspired by GFPGAN (Wang et al., 2021b), our encoder follows a UNet structure, and during its up-sampling process, we use a StyleGAN structure. We generally keep the same setting as GFPGAN, except that our encoder maps the original image from 315125122 to 3x32 x.32x256xx2 256 to build a tri- tri-planes feature space. We only modified the input and output layers to achieve this. In the experiment, information from the input image during the down-sampling process, and then generate mutually correlated planes during the up-sampling process. In order enhance the robustness of the encoder and adapt to arbitrary real-world inputs, we applied affine transformations to align the input 2D) images using estimated head poses. Since we utilize separate expression feature field encoded with PEF, the canonical feature space here lacks complete semantics on its own. Therefore, while many works based on tri-planes restrict the canonical feature space to have neutral expressions, here we do not impose any restrictions and train the encoder from scratch in an end-to-ent manner.
observed that this structure effectively integrate global
3.2 POINT-BAR BASED EXPRESSION FIELD
In this section, we will introduce how to build a controllable expression field based on point clouds. Many methods for head avatars rely on 3DMM parameters rendering images to generate expres- sions. However, they either have limited expressive capabilities when directly using 3DMM param- eters lose details due excessive encoding and decoding processes. Inspired by Point-NeRF (Xu et al., 2022), we directly use the point cloud from 3DMM to construct spint-based expression field, thereby avoiding procpeccssiing and retaining expression details much possible.
4

---

Published as a conference paper at ICLR 2024
Unlike attempts to reconstruct static scenes, our point-based expression field (PEF) aims to model dynamic expressions. To achieve this goal, we bind learnable weights to each FLAME vertex in the PEF. Due to the stable semantics and geometric topology of FLAME vertices, such as points representing the eyes and mouth not undergoing semantic changes across various expressions and identities, each neural point in the PEF also holds stable semanties and can be shared across different identities. During sampling features from PEF, we sample several nearest points to calculate the final feature for the sample position. If we sample the nearest points from the local region following (Xu et al., 2022), we may encounter limitations in representation capabilities, such that non-FLAME areas are modeled only in canonical feature space and parts related to expressions such as hair that are not included in FLAME may become fully rigid, making certain expressions unnatural. Therefore, we instead search for neighboring points in the entire space and use relative position encoding to provide the model with direction and distance information. Our approach liberates the representation capabilities of point features, and experiments have also confirmed that our method performs better. To achieve the synergy between the canonical feature space and PEF and harness the prior capabilities of point clouds, we remove the global pose from the FLAME pose and instead model it using the camera pose. This ensures that the point cloud is always in canonical position near the origin. Since we sample features from both feature spaces, the semantic information and 3D priors from the PEF can also undergo collaborative learning with the canonical feature space.
The overall process of our PEF is as follows: for any given query 3D position x during the NeRF sampling process, we retrieve its nearest K points and obtain their corresponding features fa and positions pt. Then, we employ linear layers to regress the features for each point, and finally combine these features based on positional weights, shown in Eq. 1:
K
Wi Σラ where Ipp is the linear layers and Ppos is the frequency positional encoding function. During this process, the position of point p; changes as the FLAME expression parameters change, creating a dynamic expression feature field. This allows the FLAME to directly contribute to the NeRF feature space, avoiding the loss of information introduced by excessive processing. Due to the decoupling of the canonical tri- uilplanes and PEF, we only create the canonical tri-planes once during inference, and the speed of PEF will affect the inference speed. Thanks to the efficient parallel nearest neighbor query, the PEF process can be completed quickly, greatly improving the speed of inference.
Seap,z
Lp[[é, (ps - x;), where W;- Wi
(1)
Pi
3.3 MULTI TRI-PLANES ATTENTION
Based on the aforementioned modules, we can obtain animatable high-fidelity results. However, since the source image can be arbitrary, this introduces some challenging scenarios. For example, there may be occlusions in the source image, the eyes in the source image may be closed while the desired expression requires open eyes. In this situation, the model may generate illusions based on statistically average eye and facial features, but these illusions may be incorrect. Although this image cannot produce the truth about missing parts, we may have other images that supplement the missing parts. To achieve this goal, we have implemented an attention-based module to fuse the tri-planes features of multiple images, which is called Multi Tri-planes Attention (MTA).
Our MTA uses learnable tri- plane to query multiple tri tri-planes from different images, generating weights for feature fusion, as shown in Eq. 2:
N
Wi E(1,, where w. = (B()),, wy
P
where T; is the input image, N is the number of input images, E is the canonical encoder, I. and Tak are the linear layers to generate queries and keys, and Q is the learnable query tri-planes.
Through our experiments, we have demonstrated that our MTA effectively enhances performance and completes missing information in one-shot inputs, such as pupil information and the other half of the face in extreme pose variations. During training, we use two random frames input and one frame as the target, while during inference, our MTA can accept any number of images as input. Furthermore, experiments show that MTA can consistently fuse multiple tri tri-planes features from images of the same person captured at different times. Even when dealing with images of different individuals and styles, MTA can still produce reasonable results, showcasing its strong robustness.
5

---

Published as a conference paper at ICLR 2024
3.4 VOLUME RENDERING AND SUPER RESOLUTION
Given the camera's intrinsic and extrinsic parameters, we sample the rays and perform two-pass hierarchical sampling along these rays, followed by volume rendering obtain 2D results. Due to the extensive computational resources required for high-resolution volume rendering, training and testing on high resolution become time-consuming and costly. A popular solution this problem is the lightweight super-resolution module. In our work, we render low-resolution images at 128x resolution, and these low-resolution images consist of 32-dimensional feature map with the first three dimensions corresponding to RGB pixel values. The super-resolution module we use is similar to our canonical feature space encoder, and like the encoder, we train this super-resolution module from scratch in an end-to-end manner.
3.5 TRAINING STRATEGY AND LOsS FUNCTIONS
We train our model from scratch using an end-to-end training approach. By sampling original and target images from the same video, we construct pairs of images with the same identity but different expressions and poses. During the training process, our primary objective to make the reenactment images consistent with the target images. We use Lı and perceptual loss (Johnson al.. 2016; Zhang et al., 2018) on both low-resolution and high-resolution reenactment images to achieve this objective, as shown the Eq. 3:
Pree = 1r Tell + hh Tell + (p(()s((Tr)) - + lIo(Th)) (()))
(3)
where 1, is the reenactment target image, Irr and Iar are the low-resolution and high-resolution reenactment results, is the AlexNet (Krizhevsky et al., 2012) used in the perceptual loss, and Xp the weight for the perceptual loss.
Additionally, we add a density- density-based norm loss as shown in the Eq. 4:
Loorm ||du|12.
where dn is the density used in volume rendering (Mildenhall et al., 2020). This loss encourages the total density of NeRF to be low possible, thereby encouraging reconstructions that closely adhere to the actual 3D shape and avoid the appearance of artifacts. The overall training objective is as:
Loverall Areree + Xnorss
(5)
where ,y and An are the weights that balance the loss.
EXPERIMENTS
In this section, we will first introduce the dataset use, the implementation details of our method, and the baselines of our work. We will then compare our method with existing approaches using a variety of metrics.
4.1 EXPERIMENT SETTING
Datasets. We use the VFHQ (Xie et al., 2022) dataset to train our model. This dataset comprises clips from various interview scenarios, and we utilized subset consisting of 8,013 video clips. From these videos, we extracted 240,390 frames to create our training dataset. During the training process, we randomly sampled frames from the same video to create pairs of images with the same identity but different expressions. One frame was used as the target for reenactment, while the others served as source images. Given that our method can accept any number of inputs, in each iteration, we sampled two inputs with a 70% probability and Regarding evaluation, we assessed our method on the VFHQ dataset (Xie et al., 2022) and the HDTF dataset (Zhang al., 2021). It's important note that our model was not fine-tuned on the HDTF dataset. In the evaluation process, we used the first frame each video as the source image, with the remaining frames targets for reenactment.
input with 30% probability.
Evaluation Metrics. We evaluated all methods on both same-identin and cross cross-identity recnact- ment tasks. For the cross-identity reenactment task, due the lack of ground truth, we evaluated the

---

Published as a conference paper at ICLR 2024
Source
StyleHeat
OTAvvarar
Ours One-in
Driving
Figure 4: Qualitative results on VFHQ (Xie al., 2022) and HDTF (Zhang et al., 2021) datasets.
The first two rows are from VFHQ and the third row is from HDTF.
Source Driving
ROME
StyleHeat
OTAvatar
Next3
HideNoRF
Ours One-
Figure 5:
Qualitative results VFHQ (Xie al., 2022) and HDTF (Zhang et al., 2021) datasets.
The first four rows are from VFHQ and the last row is from HDTF.
cosine similarity of identity embeddings (CSIM) based on ArcFace (Deng et al., 2019) between the reenacted frames and source images to assess identity consistency during the reenactment task. We also used the Average Expression Distance (AED) and Average Pose Distance (APD) metrics based on (Danecek et al., 2022) to assess the accuracy of expression and pose driving. In the same-identity reenactment task, where ground truth frames are available, in addition to the aforementioned met- rics, evaluate PSNR, SSIM, L1, and LPIPS metrics between the reenacted frames and ground truth frames. We also calculated the Average Key-point Distance (AKD) based on (Bulat Tz- imiropoulos, 2017) another reference for expression reenactment accuracy.
Implementation Details. Our framework is built upon the Py PoTrh framework (Paszke et al., 2017), and during the training process, employ the ADAM (Kingma & Ba, 2014) optimizer with learn-

---

Published as a conference paper at ICLR 2024
Quantitative results on the VFHQ (Xie et al., 2022) dataset. For fair comparison, we compare one-shot results using the first frame input. Ours Two-in uses both the first and last frames. Entries in green are the best ones in a one-shot setting.
Table 1:
Method
Self Reensetinent
Cross-Id Reenactment
FSNKT
SSIMF
LPIPS]
CSIM
L11
AED.
AFD]
AKD]
CSIM
AED]
AFD]
ROME (Khakhuliin et al., 2022)
19.88
0.735
0.237
0.679
0.060
0.497
0.017
4.53
0.531
0.936
Stylefleat (Yin et al., 2022)
19.95
0.738
0.251
0.603
0.065
0.593
0.024
5.30
0.506
0.961
0.038
OfAvvatar (Ma et.,,
18.10
0.600
0.346
0.660
0.092
0.734
0.035
6.05
0.514
0.962
0.059
Next3D (Sun et 2023)
19.95
0.656
0.281
0.631
0.066
0.727
0.026
5.17
0.482
0.996
0.036
HideNeRF (Li ol., 2023a)
20.07
0.745
0.204
0.794
0.056
0.521
0.031
5.33
0.558
1.024
0.044
Ours One in
22.08
0.765
0.177
0.789
0.039
00.434
0.017
3.53
0.558
0.910
0.034
Our Two-in
22.86
0.779
0.169
0.771
0.035
0.411
0.017
3.44
0.551
0.907
0,034
Table 2: Quantitative results on the HDTF (Zhang et al., 2021) dataset. For a fair comparison, we compare one-shot results using the first frame input. Ours Two-in uses both the first and last frames. Entries in green are the best ones in a one-shot setting.
Method
Sclf Recnactment
Cross-Id Reonectiment
PSNKT
SSIME
LPIPS]
CSIMF
L1,
AED
APD]
AKD]
CSIMI
AED]
APD]
ROME (Khakhullin al., 2022)
20.84
0.722
0.176
0.781
0.044
0.540
0.012
3.93
0.721
0.929
0.017
StyleHeat (Yin et al., 2022)
21.91
0.772
0.210
0.705
0.045
0.527
0.015
3.69
0.666
0.902
0.027
OTAvvatar (Ma at., 2023)
20.50
0.695
0,241
0.765
0,064
0.681
0.020
5.15
0.699
1,047
0.034
Nex33 (Sun al. 2023)
20.35
0.723
0.217
0.730
0.048
0.644
0.022
4.19
0.622
1.014
0.026
HideNeRF (Li al., 2023a)
21.38
0.803
0,147
0.907
0.038
0.499
0.027
4.33
0.803
0.032
One-in
24.21
0.029
0.427
0.012
3.06
Ous Two-in
25.36
0.849
0.122
0.851
0.026
0.406
0,012
3.01
0.769
0.837
0.021
ing rate of 1.0e-4. We conducted training on 2 NVIDIA Tesla A100 GPUs, with a total batch size of 8. During the training process, our PEF searches for the nearest K=8 points, while MTA selects two frames as source images. Our approach employs an end-to-end training methodology. The training process consists of 150,000 iterations and the full training process consumes approximately 50 GPU hours, showing its resource utilization efficiency. During the inference time, our method achieves 15 FPS when running on A100 GPU. More details can be found in the supplementary materials.
4.2
MAIN RESULTS
Baseline Methods. We compared our method with five oate-of-the-an existing methods, including StyleHeat (Yin et al., 2022) (2D-based warping), ROME (Khakhulin et al., 2022) (mesh-based), OTAvatar (Ma et al., 2023), Next3D (Sun et al., 2023; Roich al., 2021) (based on NeRF and 3D generative models), and HideNeRF (Li et al., 2023a), which is most similar to setup. All results were evaluated using official code implementations.
Self- Relf-Rennatment Results. We begin by evaluating the synthesis performance when the source and driving image are the same person. Tab. 1 and Tab. 2 show the quantitative results on VFHQ approach exhibits significant advantage over other state-
and HDTF, respectively. Notably, of-the-art methods in terms of both synthesis quality metrics (PSNR, SSIM, LPIPS, and LI) and expression control quality metries (AED and AKD). Qualitative results on VFHQ and HDTF are vi- sually demonstrated in Fig. 4. These results showcase that our method not only excels in synthesis quality but also captures subtle expressions, as exemplified by the surprised expression in the first row and the angry expression in the second row. Importantly, our model achieved these results with- out any training or fine-tuning on the HDTF dataset, thus demonstrating the robust generalization capability of our approach.
Cross-Identity Reenactment Results. We also evaluate the synthesis performance when the source and the driving images contain different persons. Tab. and Tab. 2 show quantitative results, and Fig. 5 showcases qualitative results. Due to the absence of ground truth data, quantitative evaluation of synthesis performance is not feasible, but the qualitative results evident that our method excels in expression control. These results show the efficacy of our approach in scenarios where the source and driving images are from different individuals.
Multiple images input. In addition to quantitative results, we further illustrate the advantages of multi-input methods in challenging scenarios, shown in Fig. 6, such as closed eyes and signif- icant pose variations. The results demonstrate that employing multiple inputs can further enhance synthesis quality while maintaining precise expression control.
8

---

Published as a conference paper at ICLR 2024
Figure 6: Qualitative results of multi-inputs. In each image, the left side shows imput frames, while the right side displays reenactment frames and driving frames. It can be observed that using multiple inputs enhances the performance, especially in cases of closed eyes and occlusions.
Table 3: Ablation results on the vFHQ dataset. Entries in green are the best ones.
Metbod
PSNKT
SSIMT
LPIPS]
CSIM
L11
AED]
APD]
AKD]
w/o PEF
22.01
0.762
0.186
0.166
0,040
0.576
0.020
4.30
w/o global sample
21.58
0.760
0.194
0.765
0.039
0.518
0.019
3.96
point cloud 2000
21.95
0.761
0.193
0.750
0,040
0.497
0.020
3.86
query points
22.04
0.762
0.192
0.751
0.039
0.514
0.020
3,90
Ours One-in
22.08
0.765
0.177
0.789
0.039
0.434
0.017
3.53
mean Two-in
22.75
0,776
0.190
0.726
0.036
0.452
0.019
3.68
0.191
mean Four-in
23.16
0.783
0.194
0.716
0.035
0.449
0.018
3.64
Ours Two-in
22.86
0.779
0.169
0.771
0.035
0.411
0.017
3,44
Ours Three-in
23.27
0.788
0.165
0.772
0.033
0.403
0.016
3.41
Ours Four-in
23.49
0.792
0.164
0.773
0,032
0.400
0.016
3.41
4.3 ABLATION STUDIES
Effectiveness of Point-based Expression Field. To validate the effectiveness of our proposed PEF, we provide FLAME expression parameters directly as baseline for comparison (w/o PEF in Tab. 3). This method was applied in NeRFace and demonstrated expression control capability. The improvements in AED and AKD clearly indicate that our PEF significantly enhances expression control. We also tried sampling points in a local area with maximum distance of 1/128 instead of global sampling. The results are shown as w/o global sample Tab. 3. The results show that our global sampling enhances the details in expressions and the quality of synthesis.
Effectiveness of Multi Tri-planes Attention. To validate the effectiveness of our proposed MTA, established naive mean-based baseline that averages the tri tri-planes of multiple images obtain a merged plane. Table 3 shows the results. We observe that our MTA exhibits better synthesis performance, which attribute to MTA's ability avoid feature blurring caused by average fusion.
Ablation on Hypeer-arameters. We conducted experiments on the selection of We randomly selected 2,000 points from 5,023 points of FLAME, and the results in Tab. 3 show that our method can also work on sparse point clouds. This may be attributed to our PEF finding neighboring points from the entire space, which prevents sparse sampling issues. We also reduced the number of query neighbors K from 8 to 4, and the results indicate that our method has some robustness to the number of neighboring points.
CONCLUSION
In this paper, we have introduced novel framework for generalizable and precise reconstruction of animatable 3D head avatars. Our approach reconstructs the neural radiance field using only one or a few input images and leverages a point-based expression field to control the expression of synthesized images. Additionally, we have introduced an attention-baa based fusion module to utilize information from multiple input images. Ablation studies suggest that the proposed Point-based Expression Field (PEF) and Multi Tri-planes Attention (MTA) can enhance synthesis quality and expression control. Our experimental results also demonstrate that our method achieves the most precise expression control and mute-offthe oa-t-off-hee-nt art synthesis quality multiple benchmark datasets. We believe that our method has a wide range of potential applications due to its strong generalization and precise expression control capabilities.

---

Published as a conference paper at ICLR 2024
6 ETHICS STATEMENT
Since our framework allows for the reconstruction and reenactment of head avatars, has a wide range of applications but also carries the potential risk of misuse, such as using create fake videos of others, violating privacy, and spreading false information. We are aware of the potential for misuse of our method and strongly discourage such practices. To this end, we have proposed several plans to prevent this technical risk:
We will add conspicuous watermark to the synthesized video so that viewers can easily identify whether the video was synthesized by the model. This will significantly reduce the cost for viewers to identify the video and reduce the risk of abuse.
We limit the identity of the target speaker to virtual identities such as virtual idols, and prohibit the synthesis of real people without formal consent. Furthermore, synthetic videos may only be used for educational or other legitimate purposes (such as online courses) and any misuse will be subject liability via the tracking methods present in the point.
We will also inject invisible watermarks into the synthesized video to store the IP of the video producer, so that the video producer must consider the potential risks brought by the synthesized video. This will encourage video producers to proactively think about whether their videos will create ethical risks and reduce the possibility of creating abusive videos.
To summarize, as a technology designer, we come up with strict licenses and technologies to pre- vent abuse of our GPAvatar, talking face reconstruction system. We think more efforts from gov- ermments, society, technology designers, and users are needed to eliminate the abuse of deepfake. Besides, we hope the video maker is aware of the potential risks and responsibilities when using the talking face generation techniques. We believe that, with proper application, our method has the potential to demonstrate significant utility in various real-world scenarios.
7 REPRODUCIBILITY STATEMENT
Here summarize the efforts made ensure the reproducibility of this work. The model architec- tures and training details are introduced in Appendix I. and we also release the code for the model https::/github..com. http:ttth...........eeeeeeeoeoee Noom/xgcchu/GPAAwe/ The data processing and evaluation details are introduced in Appendix A.2 and Appendix A.3.
ACKNOWLEDGMENTS
This work was partially supported by IST Moonshot R&D Grant Number JPMJPS2011, 1, CREST Grant Number JPMJCR2015 and Basic Research Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo. This work was also partially supported by JST, the establishment of university fellowships towards the creation of science technology innovation, Grant Number JP- MJFS2108.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu. Rignerf: Fully controllable neural 3d portraits. Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 20364-20373, 2022.
ShahRukh Athar, Zhixin Shu, and Dimitris Samaras, Flame-in-nerf. Neural control of radiance fields for free view face animation. In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG), pp. 1-8. IEEE, 2023.
Ziqian Bai, Feitong Tan, Zeng Huang, Kripasindhu Sarkar, Danhang Tang, Di Qiu, Abhimitra Meka, Ruofei Du, Mingsong Dou, Sergio Orts-Esco,,no, et al. Learning personalized high quality volu- metric head avatars from monocular rgb videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16890-16900, 2023.
Volker Blanz and Thomas Vetter. morphable model for the synthesis of 3d faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 1999.
Adrian Bulat and Georgios Tzimiropoulos. How far we from solving the 2d & 3d face alignment problem? (and dataset of 230,000 3d facial landmarks). In IEEE International Conference on Computer Vision (ICCV), 2017.
Radek Danecek, Michael Black, and Timo Bolkart. EMOCA: Emotion driven monocular face capture and animation. In Conference on Computer Vision and Pattern Recognition (CVPR), pp- 20311-20322, 2022.
Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2019.
Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. Proceedings of the 30th ACM International Conference on Multimedia, 2022.
Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. Learning an animatable detailed 3D face model from wn-the-wild images. ACM Transactions on Graphics (ToG), Proc. SIGGRAPH, pp. 88:1-88:13, 2021.
Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Luthi, Sandro Schönbom, and Thomas Vetter. Morphable face models-an open framework. In 2018 13th IEEE International Conference Automatic Face & Gesture Recognition (FG 2018), pp. 75-82, 2018.
Yudong Guo, Keyu Chen, Sen Liang. Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Au- dio driven neural radiance fields for talking head synthesis. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5784-5794, 2021.
Yang Hong, Bo Peng. Haiyao Xiao, Ligang Liu, and Juyong Zhang, Headnerf: A real-time nerf- based parametric head model. In IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2022.
Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pp. 694-711. Springer, 2016.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 4401-4410, 2019.
Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Egor Zakharov. Realistic one-shot mesh-based head avatars. In European Conference Computer Vision (ECCV), 2022.

---

Published as a conference paper at ICLR 2024
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 2012.
Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning model of facial shape and expression from 4D scans, ACM Transactions on Graphies, (Proc. SIGGRAPH Asia), pp. 194:1-194:17, 2017.
Weichuang Li, Longhao Zhang, Dong Wang, Bin Zhao, Zhigang Wang, Mulin Chen, Bang Zhang, Zhongjian Wang, Liefeng Bo, and Xuelong Li. One-shot high-fidelity talking- head synthesis with deformable neural radiance field. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023a.
Xueting Li, Shalini De Mello, Sifei Liu, Koki Nagano, Umar Iqbal, and Jan Kautz. Generalizable one-shot neural head avatar. Arxiv, 2023b.
Zhiyuan Ma, Xiangyu Zhu, Guojun Qi, Zhen Lei, and Lei Zhang. Otavatar: One-shot talking face avatar with controllable tri-plane rendering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Taneik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. IEEE Interna- tional Conference on Computer Vision (ICCV), 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Gold- man, Ricardo Martin- Martin-Brualla, and Steven M. Seitz. Hypernerf: A -igher-dimensional representa- tion for topologically varying neural radiance fields. ACM Trans. Graph., 2021b.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. In 2009 sixth IEEE international conference on advanced video and signal based surveillance, pp. 201-301, 2009.
Yurui Ren, Ge Li, Yuanqi Chen, Thomas Li, and Shan Liu. Pirenderer: Controllable portrait
image generation via semantic neural rendering. In IEEE International Conference on Computer Vision (ICCV), 2021.
Daniel Roich, Ron Mokady, Amit Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Trans. Graph., 2021.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances Neural Information Processing Systems (NeurIPS), 2019.
Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, and Yebin Liu. Next3d: Generative neural texture rasterization for 3d-aware head avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20991-21002, 2023.
Jaxiaag Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition. arXiv preprint 2rXiv:2211.12366 2022.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In IEEE International Conference on Computer Vi- sion (ICCV), 2021.
12

---

Published as a conference paper at ICLR 2024
Alex Trevithick, Matthew Chan, Michael Stengel, Eric R. Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, and Koki Nagano. Real-time radiance fields for single-image portrait view synthesis. In ACM Transactions Graphics (SIGGRAPH), 2023.
Ting-Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021a.
Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021b.
Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and benchmark for video face super-resolution. In The IEEE Conference on Computer Vision and Puttern Recognition Workshops (CVPRW), 2022.
Qiangeng Xu, Zexiang Xu, Julien Philip. Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and UIrich Neu- mann. Point-nerf: baset neural radiance fields. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp.
2022.
Yuelang Xu, Hongwen Zhang, Lizhen Wang, Xiaochen Zhao, Han Huang, Guojun Qi, and Yebin Liu. Latentavatar: Learning latent expression code for expressive neural head avatar. arXiv preprint ar arXiv:2305.00 190, 2023.
Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In European Conference on Computer Vision (ECCV), 2022.
Heng Yu, Koichiro Niiumma, and László A Jeni. Confies: Controllable neural face avatars. In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG), pp. 1-8. IEEE, 2023a.
Wangbo Yu, Yanbo Fan, Yong Zhang. Xuan Wang, Fei Yin, Yunpeng Bai, Yan-Pei Cao, Ying Shan, Yang Wu, Zhongqian Sun, et al. Nofa: Nerf-based one-shot facial avatar reconstruction. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-12, 2023b.
Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. Fast bi-layer neural synthesis of one-shot realistic head avatars. In European Conference on Computer Vision (ECCV), 2020.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8652-8661, 2023.
Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face gen- eration with a high-resolution audio-visual dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C Bühler, Xu Chen, Michael J Black, and Otmar Hilliges. Im avatar: Implicit morphable head avatars from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13545-13555, 2022.
Wojciech Zielonka, Timo Bolkart, and Justus Thies. Instant volumetric head avatars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4574 4584, 2023.
13

---

Published as a conference paper at ICLR 2024
A REPRODUCIBILITY
A.1 MORE IMPLEMENTATION DETAILS
Specifically, our canonical feature encoder takes the original image of 512 512 as input. We obtain the style code through 4 groups of ResBlock down-sampling, use 3 groups of ResBlock up- sampling to obtain the conditions, and then use StyleGAN (Karras et al., 2019) to output the x 32 x 256 x 256 tri-planes based style code and conditions. In the point-based expression field, we assign a 32-dim feature to each point. Since FLAME (Li et al., 2017) contains 5023 points, the total point features size is 5023 x 32. During sampling in PEF, we select the nearest 8 points, compute 39-dim relative position code for each point, and use two fully connected layers to map the 71-dim features to 32-dim. In the multi tri-planes attention module, the query and key for attention calculation, and the input, output, and hidden layer dimensions are all set to 32. After obtaining 32-dim features from the canonical feature space and expression field, we use three dense layers to map the feature from 64-dim to 128-dim. Subsequently, a single linear layer is employed predict density, and two linear layers are used to predict RGB values. Finally, we employ a super-reoolution network, similar to the encoder, to map the image from 32 x 128 x 128 to 512 x 512 dimensions. We also provide code for the model in the supplementary materials for reference.
employ three linear layers to map
A.2 MORE DATA PROCESSING DETAILS
We use 8,013 video clips from the VFHQ dataset (Xie et al., 2022) for training, and we uniformly sampled 30 frames from each video clip to ensure that the expressions and poses in each frame were as diverse as possible, resulting in a total of 240,390 frames. We cropped the heads and shoulders from the videos, extracted the 3DMM parameters for each frame (including identity, expression, and camera pose) with (Danecek et al., 2022) and further refined the pose with (Bulat & Tzimiropoulos, 2017). Finally, we resized all these images 512 x 512 pixels.
For the HDTF (Zhang et al., 2021) dataset, we followed the training-testing division in OTAvatar. We conducted a uniform lime-based sampling. selecting 100 frames from each of the 19 videos, thereby creating a test split encompassing 1900 frames. As for the VFHQ dataset, we employed similar approach, uniformly sampling 60 frames from each of the 30 videos. This method ensured that all parts of each test video were sampled as thoroughly as possible.
A.3 MORE EVALUATION DETAILS
We conducted comparisons with ROME (Khakhulin et al., 2022), StyleHeat (Yin et al., 2022), OTA- vatar eM al., 2023), Next3D (Sun al., 2023), and Hide (Li al., 2023a) 20233) using their official implementations. Since NOFA (Yu et al., 2023b) does not currently provide an official implementa- tion, and GOAvatar (Li al., 2023b) a parallel work, is challenging make a correct and fair comparison between these two. Additionally, as Next3D (Sun et al., 2023) has not yet provided a formal inversion implementation, we integrated PTI (Roich al., 2021) for reconstruction within it.
For each method, we utilized the official data pre- processing scripts to obtain their respective input frames, driving frames, and result frames. For all methods, we aligned the facial regions to uniform size and then resized them to 512x512. It's important to note that the same alignment parameters were applied both the driving frames and result frames to ensure their correspondence. Subse- quently, we computed all metrics on the aligned frames to ensure a fair comparison. Furthermore, as most methods primarily focus on the facial area, our approach actually encompasses a larger region, including parts of the shoulders. During alignment, we used region closer to the face to make as few modifications as possible to the baseline method's 's results, which may result in our approach not performing optimally in some metrics.
B LIMITATIONS
Our method has some limitations. Specifically, our current FLAME- model lacks module to control the shoulders and body, resulting in limited control below the neck Currently, the position of the shoulders in the images generated by model is generally consistent with the input image.
14

---

Published as a conference paper at ICLR 2024
Additionally, for other areas not modeled by FLAME such hair and tongue, explicit control is also not feasible. Furthermore, while our aspiration is to achieve real-time reenactment more than 30 fps. current performance is pre-real-time for now (approximately 15 fps on the A100 GPU). We leave addressing these limitations for future work.
C MORE ABLATION STUDIES
Inputs
Drive
Mean Two-in
Ours Two-in
Inputs
Drive
Mean Twe-in
Ours Twe-in
Figure 7: Qualitative results on VFHQ (Xie et al., 2022) datasets. Compared to the mean baseline, our MTA preserves more details. The smaller of the inputs is provided Ours Two-in.
Drive
Ours Four-in
Inputs
Drive
Ours Four-in
Figure 8: Qualitative results VFHQ (Xie et al., 2022) datasets. With four inputs, method
produces sharp and detailed results.
To thoroughly explore design choices for the model, conducted additional ablation experiments. More Ablation with MTA. Fig. 7 presents the qualitative results compared to the naive mean baseline. From the qualitative results, the eyes and facial features, leading to decrease in performance, which aligns with the observations from the quantitative results. Fig. 8 displays the qualitative results of our method when using four images input. It can be observed that using four images input does not result in detail smoothing or loss and has a quite good performance.
can observe that the mean baseline excessively smooths
Visualization of attention map in MTA. In order to better evaluate the performance of MTA, we visualized the attention map. As shown Fig.9, we can see that the model can pay attention to different parts of the face very well. When the input includes left and right faces, the model can provide attention to both sides to achieve more complete modeling of the whole face. At the same time, the model also pays reconstructed correctly. These visualization results are consistent with our expectations and show the advantages of MTA in fusing multiple image inputs.
attention to the open-eyes input to ensure that the eye area is
15

---

Published as a conference paper at ICLR 2024
Figure 9: Visualization of attention map in our MTA module. The image is normalized by affine transformation. Red represents higher attention and black represents lower attention. MTA can pay attention to the left and right faces in different inputs, and also pay more attention to the eyes-open photos.
D PRELIMINARIES OF FLAME
We the geometry prior from the FLAME (Li al., 2017) model, a 3D morphable model known for its geometric accuracy and versatility. It extends beyond static facial models by incorporating expressions, offers precise control over facial features, and is represented parametrically. FLAME finds applications in facial animation, avatar creation, and facial recognition due to its realistic ren- dering capabilities and flexibility. The FLAME model represents the head shape in the following way:
TP(3,0, 0, =T T + BP(( BE( E),
6
where 7 is a template mesh BS(33 S) a shape blend-shape function to account for identity- related shape variation, BP( P) is corrective pose blend-shape to correct pose deformations that cannot be explained solely by linear blend skinning, and expression blend-shaper (E E)) is used to capture facial expressions.
E BLENDING MULTIPLE IDENTITIES
We also attempted to synthesize results using different individuals and styles as inputs. As shown in Fig. 10, even in the case of such diverse inputs, our method demonstrated robustness, producing reasonable results while combining features from different persons in the images.
Inputs
Reenactment
Inputs
Reenactment
Inputs
Reenactment
Inputs
Reenactment
Figure 10: Blend results from in-the-wild images. The smaller images are the driving frames.
16

---

Published as a conference paper at ICLR 2024
F MORE QUALITATIVE RESULTS
In this section, we showcase more visual results of our method for cross-identity reenactment. Fig. 11 showcases additional results of method. It's worth noting that, for fair comparison with other methods, we standardized the evaluation details across all methods, as stated in the evaluation specifics in Appendix A.3. Our method can also encompass more non- non-facial regions, as illustrated in Fig. 11. Fig. 12 displays the results on VFHQ, and Fig. 13 presents results on HDTF, respectively. We also provide supplementary video show more dynamic results.
17

---

Published as a conference paper at ICLR 2024
M
Inputs
Drive
Ours One-in
Ours Two-in
Inputs
Drive
Ours One-in
Ours Two-in
Figure 11: Qualitative results on VFHQ (Xie et al., 2022) datasets. The smaller of the two source inputs is provided to Ours Two-in.
18

---

Published as a conference paper at ICLR 2024
Source / ROME Driving
StyleHeat
OTAvatar
Next3D
HideNeRF
Ours One-in
Ous Two-in
Qualitative results on VFHQ (Xie et al., 2022) datasets. The smaller of the two source inputs is provided to Ours Two-in.
Figure 12:
19

---

Published as a conference paper at ICLR 2024
Source ! ROME Driving
StyleHeat
OTAvatar
Next3D
HideNeRI
Ours One-in
Ours Two-in
Qualitative results on HDTF (Zhang et al., 2021) datasets. The smaller of the two source inputs is provided Ours Two-in.
Figure 13:
20


Reviews:
[{"review_id": "u9kmW9LpNF", "reviewer": "Reviewer_MrBB", "summary": "In this paper, a novel framework named GPAvatar is introduced, designed to reconstruct 3D head avatars seamlessly from one or multiple images in a single forward pass. The key novelty lies in the incorporation of a dynamic point-based expression field, guided by a point cloud, to intricately and efficiently capture facial expressions. The authors present the concept of a dynamic Point-based Expression Field (PEF), enabling accuracy and control of expressions across different identities. Additionally, they introduce a Multi Tri-planes Attention (MTA) fusion module, capable of handling a varied number of input images with precision.", "questions": "- It is not clear to me how the model captures the subtle information such as closed eyes ? \n- why the normalization of the weight wi in the equation is required?\n- Several terms in the equation (page 3) I_t= R (....) are not defined. What R means?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall, the paper is well-organized and easy to follow. The motivation is clear. The figures and tables are informative.\n\n- Experimental results demonstrate that the proposed method achieves the most precise expression control and state-of-the-art synthesis quality (StyleHeat, ROME, OTAvatar, and Next3D) (based on NeRF and 3D generative models)n on multiple on VFHQ and HDTF benchmark datasets.", "weaknesses": "- The model proposed has overall more trainable parameters compared to baseline models, which could potentially bring in some unfairness during comparison with other works.\n- No discussion about the limitations of the approach?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DsijxVgigI", "reviewer": "Reviewer_q9Mp", "summary": "This paper proposes a method that reconstructs 3D head avatars from images and synthesizes realistic talking head videos. It takes as input a single or a small number of face images and reconstructs 3D head avatars in a single forward pass. It extends the formulation of NERFS and proposes a dynamic point-based expression field that is driven by a point cloud, motivated by the need to have an accurate control of facial expressions. In addition, the proposed method adopts a Multi Tri-planes Attention (MTA) fusion module that facilitates the 3D representation of the scene and the incorporation of information from multiple input images. The proposed method is compared with several SOTA methods and achieves promising results.", "questions": "- Section 3.2: the paper fails to clearly explain how the expression information affects the process of building a point-based expression field.\n\n- Section 3.3: the paper provides insufficient details about about how the canonical encoder is defined and built.\n\n- Figure 4: For several columns with results, it is unclear which is the corresponding method. There is one column more than the number of methods in the caption and one column more than the columns of Figure 5. This creates confusions. \n\n- Equation (1): the definition of w_i does not seem to make sense. Is there a missing norm in the denominator?\n\n- After Equation (2): N is refereed to as the \"input number\", but apparently it should be referred to as the number of input images", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The proposed method achieves promising results and the supplementary videos show that the videos synthesized by the proposed method are in general realistic and visually pleasing. \n\n+ The experimental evaluation is detailed and systematic. The proposed method is compared with several recent SOTA methods that solve the same problem.", "weaknesses": "- The presentation in several parts of the paper, especially in the methodology, is unclear and needs several clarifications. See detailed comments in Questions below. \n\n- The following paper is not cited, despite the fact that it is very closely related in terms of methodology:\n\nAthar, S., Shu, Z. and Samaras, D., 2023, January. Flame-in-nerf: Neural control of radiance fields for free view face animation. In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG) (pp. 1-8). IEEE.\n\nThe above un-cited paper also uses a FLAME-based representation of the 3D face and extends the formulation of NERFS to achieve realistic face animation with expression control. The similarities and differences with the proposed method are not discussed and the Flame-in-nerf is not included in the comparisons of the experimental section.  This raises concerns in terms of the real novelty and contributions of the proposed method. \n\n- Furthermore, there are also several other closely related works that are not cited. For example: \n\nJiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition. arXiv preprint arXiv:2211.12368, 2022.\n\nYudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In ICCV, pp. 5784–5794, 2021.\n\nYu, H., Niinuma, K. and Jeni, L.A., 2023, January. CoNFies: Controllable Neural Face Avatars. In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG) (pp. 1-8). IEEE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fr9Rkz9foM", "reviewer": "Reviewer_mo8c", "summary": "The authors proposed a one-shot/few-shot NeRF-based talking face system. The contributions are twofold: (1) a point-based expression field (PEF) for 3D avatar animation; (2) a multi triplanes attention (MTA) that supports multiple images as input to handle hard cases like occlusion or closed eyes. I like some ideas in this paper. This is a overall well-written paper and is easy to follow. The experiment shows good performance over previous baselines. \n\nHowever, the identity similarity in the demo video is not as good as previous one-shot 3D talking face methods (such as HiDe-NeRF). Besides, I'm also curious about the performance of this method under large head poses, which is not revealed in the demo.", "questions": "- In  PEF, the expression feature of facial part can be queried from the FLAME mesh, but the non-facial part, such as hair, clothes, and background is not modeled by FLAME. The authors said \"we instead search for neighboring points in the entire space\", but it is not clear how it bundles the non-facial part in the 3D space with the learnable features.\n- The identity similarity in the demo video is not as good as previous one-shot 3D talking face methods (such as HiDe-NeRF), what's the cause?\n- The head movement in the provided video is quite gentle. Is there any demo where the head pose is larger (such as side view)? Since one of the biggest advantage of 3D methods over the traditional 2D methods is the good quality under a large view angle, I think this is necessary for the reviewer to assess the performance of a 3D-based work.\n- Could you provide the visualization of the attention weights in the multi-reference setting? I'm also curious about the scalability of the MTA, how is the attention map looks like under the two-in/five-in/ten-in ?\n- In the demo video, the avatars are driven by audio, it is better to illustrate the way used to obtain the facial expression.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I like the idea of PEF since it well utilizes the geometry prior of FLAME to help learn the avatar animation in the 3D space. \n- Also, it is the first one-shot 3D talking face paper that focuses on the few-shot setting.\n- The paper is well-written and is easy to follow.", "weaknesses": "- the PEF could well handle the segment modeled by FLAME (such as head and torso), but it cannot handles other parts, such as hair and clothes. See question 1.\n- The identity similarity in the demo video is worse than some baseline (HideNeRF).\n- The image quality can be improved. For instance, in Figure 1, the predicted images in the second column seems blurry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)