Generalizable and Animatable Gaussian Head Avatar
Authors: Xuangeng Chu
Tatsuya Harada
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Gaussian Splatting
Head Avatar
Dynamic 3D Gaussian Splatting
Primary area: machine_vision

Abstract: In this paper, we propose Generalizable and Animatable Gaussian head Avatar (GAGA) for one-shot animatable head avatar reconstruction.
Existing methods rely on neural radiance fields, leading to heavy rendering consumption and low reenactment speeds.
To address these limitations, we generate the parameters of 3D Gaussians from a single image in a single forward pass.
The key innovation of our work is the proposed dual-lifting method, which produces high-fidelity 3D Gaussians that capture identity and facial details.
Additionally, we leverage global image features and the 3D morphable model to construct 3D Gaussians for controlling expressions.
After training, our model can reconstruct unseen identities without specific optimizations and perform reenactment rendering at real-time speeds.
Experiments show that our method exhibits superior performance compared to previous methods in terms of reconstruction quality and expression accuracy.
We believe our method can establish new benchmarks for future research and advance applications of digital avatars.

Generalizable and Animatable Gaussian Head Avatar
Xuangeng Chu The University of Tokyo xuangeng- chu@mi t. w.t.uoooo.aaoo jc.jp
Tatsuya Harada The University of Tokyo RIKEN AIP
harada0mi t. u-tooooo..cc ac.
GAGAvatar
Source Images
Driving
Result Images
Figure 1: Our method can reconstruct animatable avatars from a single image, offering strong generalization and controllability with real-time reenactment speeds.
Abstract
In this paper, we propose Generalizable and Animatable Gaussian head Avatar (GAGAvatar) for one-shot animatable head avatar reconstruction. Existing methods rely on neural radiance fields, leading to heavy rendering consumption and low reenactment speeds. To address these limitations, we generate the parameters of 3D Gaussians from a single image in single forward pass. The key innovation of our work the proposed dual-lifting method, which produces high-fidelity 3D Gaussians that capture identity and facial details. Additionally, we leverage global image features and the 3D morphable model to construct Gaussians for controlling expressions. After training, our model can reconstruct unseen identities without specific optimizations and perform reenactment rendering at real-time speeds. Experiments show that our method exhibits superior performance compared to previous methods in terms of reconstruction quality and expression accuracy. We believe method can establish new benchmarks for future research and advance applications of digital avatars. Code and demos are available at https://github.com/ wtwpt//owoocccccg///a/aaaa chu/GAGAvatar.
Introduction
One-shot head avatar reconstruction has garnered significant attention in computer vision and graphics recently due to its great potential in applications such as virtual reality and online meetings. The typical problem involves faithfully recreating the source head from image while precisely controlling expressions and poses. In recent years, many exploratory methods have achieved this goal using 2D generative models and 3D synthesizers.
Some early 2D-based methods [Yin et al., 2022, Ren et al., 2021] typically combine estimated deformation fields with generative networks to drive images. However, due to the lack of neces- sary 3D constraints and modeling, these methods struggle to maintain multi-view consistency of expressions and identities when head poses change significantly. Recently, Neural Radiance Fields (NeRF) [Mildenhall et al., 2020] have shown impressive results in head avatar synthesis, providing
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

solutions using 3D synthesizers to achieve realistic details such as accessories and hair. However, some NeRF-based methods [Ma et al., 2023] require identity-speciffs training and optimization. and some methods [I. et al., 2023a, Chu et al., 2024, Deng et al., 2024a] can't render in real-time during inference, limiting their application in certain scenarios. With the emergence of 3D Gaussian splatring [Kerbl et al., 2023], some methods [Xu et al., 2024] have achieved real-time rendering. However, these methods still require specific training for each identity and fail to generalize to unseen identities, leaving the modeling of generalizable 3D Gaussian-base based head models unexplored.
To address these limitations, we introduce novel 3D ceuussian-based framework for one-shot head avatar reconstruction. Given a single image, our framework reconstructs an animatable 3D Gaussian baussian-based head avatar, achieving real-time expression control and rendering. Some examples are shown in Fig. 1. The core challenge lies in faithfully reconstructing 3D Gaussians from a single image, as a 3D Gaussian typically requires multi-view input and millions of Gaussian points for detailed reconstruction. To address this, we propose novel dual-lifting method that reconstructs the 3D Gaussians from one image. Specifically, instead of directly estimating Gaussian points from the image, we predict the lifting distances of each pixel relative to the image plane, and then map the image plane and lifted points back to 3D space based on the camera position. By predicting forward and backward lifting distances, we can form an almost closed Gaussian points distribution and reconstruct the head as completely as possible. This approach leverages the fine- fineggraiee features of the input image and significantly reduces the difficulty of predicting 3D Gaussian positions. We also utilize priors from 3D Morphable Models (3DMM) [Li et al., 2017] to further constrain the lifting distance, helping the model obtain correct 3D lifting and capture details from the source image. We then bind learnable features to the 3DMM vertices and construct expression Gaussians using image global features, 3DMM learnable features, and 3DMM point positions ensure expression control capability. Finally, we use a neural renderer to refine the splatting-renders results, producing the final reenacted image. Our model is learned from a large number of monocular portrait images and can be generalized to unseen identities after training. Experiments verify that our method performs better than previous methods in terms of reconstruction quality and expression accuracy, and achieves real-time reenactment and rendering speed.
Our major contributions can be summarized follows:
We propose GAGAvatar, which to our knowledge is the first generalizable 3D Gaussian head avatar framework that achieves single forward reconstruction and real-time reenactment.
To achieve this, we propose a dual-lifting method to lift Gaussians from a single image and introduce a method that uses 3DMM priors to constrain the lifting process.
We combine 3DMM priors and 3D Gaussians to accurately transfer expression information while avoiding redundant computations.
2 Related Work
2.1 2D-based Talking Head Synthesis
The impressive performance of CNN and Generative Adversarial Networks (GAN) [Goodfellow al., 2014, Isola al., 2017, Karras et al., 2020] has inspired many methods for direct head image synthesis using 2D networks. A popular strategy of early works is inserting the expression and head pose features of the driving image into the 2D generative network to achieve realistic and animatable image generation. For example, these methods [Zakharov et al., 2019, Burkov et al., 2020, Zhou et al., 2021, Wang al., 2023] inject latent representations of expression into the U-Net backbone or StyleGAN-like [Karras et al., 2019] generators to transfer driving expressions to reenacted images. A recent trend in 2D-based talking head synthesis methods [Siarohin al., 2019, Ren al., 2021, Drobyshev et al., 2022, Hong et al., 2022a, Zhang et al., 2023a] is to represent expressions and head poses as warp fields, performing expression transfer by deforming the source image to match the driving image. However, due to the lack of explicit understanding of the 3D geometry of head portraits, these methods often produce unrealistic distortions and undesired identity changes when there are significant pose and expression variations. Although some methods [Drobyshev et al., 2022, Wang et al., 2021a, Ren et al., 2021, Yin et al., 2022, Zhang et al., 2023b] introduce 3D Morphable Models (3DMM) [Blanz and Vetter, 1999, Paysan et al., 2009, Li et al., 2017, Gerig et al., 2018]
2

---

into the 2D framework, they still lack the ability to control the viewpoint and achieve free-viewpoint rendering. Additionally, there are some audio-driven 2D control methods [Guo et al., 2021, Tang et al., 2022, Zhang et al., 2023b], while flexible to use, cannot explicitly control facial expressions and poses, sometimes resulting unsatisfactory outcomes. In contrast, our method uses an explicit 3D representation to enable free view control and realistic synthesis even under large pose variations.
2.2 3D-based Head Avatar Reconstruction
To achieve better 3D consistency in head avatars, many works have explored using 3D representations for reconstruction. Early methods [Xu et al., 2020, Khakhulin et al., 2022] used 3DMM-based meshes [[. et al., 2017, Gerig et al., 2018] to reconstruct head avatars. Since neural radiance fields (NeRF) [Mildenhall et al., 2020] have demonstrated excellent results, many recent methods [Li et al., 2023b,a, Ma et al., 2023, Yu et al., 2023, Chu et al., 2024, Ye et al., 2024, Deng et al., 2024b.a, Park et al., 2021a, Zheng et al., 2023, Bai et al., 2023a, Ki et al., 2024] have adopted NeRF for head reconstruction. However, some approaches [Gafni et al., 2021, Park et al., 2021a, Tretschk et al., 2021, Hong et al., 2022b, Athar et al., 2022, Park al., 2021b, Gao et al., 2022, Guo al., 2021, Bai et al., 2023b, Kirschstein et al., 2023, Zheng et al., 2023, Bai et al., 2023a, Zhao et al., 2023, Zhang al., 2024] require multi-view or single-view view videos of specific identities for training, limiting generalization and raising privacy concerns due to the need for thousands of frames of personal image data. Additionally, some methods [Xu et al., 2023a, Tang et al., 2023, Sun et al., 2022, Xu et al., 2023b, Zhuang et al., 2022a, Sun et al., 2023] train generators to produce controllable head avatars from random noise, followed by inversion [Roich et al., 2022, Xie et al., 2023] for ddentity-specifff reconstruction. These methods often suffer from inversion accuracy limitations, failing to preserve the identity of the source image. There are also methods [Hong et al., 2022b, Zhuang et al., 2022b, Ma et al., 2023] to perform test-time optimization on the source image to obtain reconstructions, but the need for test-time optimization limits their applicability. To address these challenges, some works [Yu et al., 2023, Li et al., b023a,b, Ma et al., 2024a, Yang et al., 2024, Chu et al., 2024, Ye et al., 2024, Ma et al., 2024a, Deng et al., 2024b,a] focus on one-shot head reconstruction without test-time optimization. For example, GOHA| [Li et al., 2023a] learns three tri-plane features capture details. HideNeRF [Li et al., 2023b] utilizes multi-resolution tri-planes and a deformation field to generate reenactment images. GPAvatar [Chu et al., 2024] uses a point-based expression field and multi trinp attention module to reconstruct head avatars. Real3DPortrait [Ye et al., 2024] generates a tri-plane from images and adds motion adapters get reenactment images. CVTHead [Ma et al., 2024a] reconstructs head avatars using point-based neural rendering and a vertex-feature transformer. Portrait4D [Deng et al., 2024b] learns dynamic expression tri-plane from multi-view synthetic data, while Portrait4D-v [Deng et al., 2024a] learns from pscudo multi-view vidcos, addressing the lack of real video training in Portrait4D. However, these NeRF-based methods often face rendering speed limitations, preventing real-time application. Methods [Xu et al., 2024, Li et al., 2024, Hu et al., 2023, Wang et al., 2024a, Ma et al., 2024b, Wang et al., 2024b] utilizing 3D Gaussian splatting[ 2023] achieve excellent performance and rendering speed but require video data for Kerbl et al., training. lacking generalization capabilities. In this paper, we propose shot 3D Gaussian head avatar reconstruction method based on the dual-lifting method. Our method can generalize to unseen identities, achieves real-time rendering, and surpasses previous works in image quality.
3 Method
An overview of the reenactment process of our method is shown in Fig. 2. Given a source image I,, we first use DINOv2 [Darcet et al., 2023, Oquab et al., 2023] to extract global and local features. Using the local features, we apply our proposed dual-lifting methods to predict the parameters and positions of two 3D Gaussians. Simultancously, we assign learnable parameters to each vertex of the 3DMM [Li et al., 2017] model and predict another expression Gaussians using the combination of the global feature and vertex features. We directly use the vertex positions of the 3DMM model as the positions for expression Gaussians. Finally, we combine these 3D Gaussians and perform splatting to produce coarse result image I. with the expression and pose of driving image Id, which is then further refined through a neural renderer to obtain the fine result image I,-
In the following subsections, we describe the reconstruction branch based on dual-lifting in Sec. 3.1, explain the expression modeling and control branch in Sec. 3.2, and detail our neural renderer in Sec. 3.3, Finally, we describe our lifting distance loss and the training objectives in Sec. 3.4.
3

---

Expression Branch
Driving Part
3DMM (Offline)
ENLP
Peez wjapas
Poines Position
Points
Driving Image Fz
Splatting
Enender
Image
Image ly
Reconstruction Branch
*
Warp
DINOV2
Pasal
Image
Figure 2: Our method consists of two branches: reconstruction branch (Sec. 3.1) and expression branch (Sec. 3.2). We render dual-lifting and expressed Gaussians to get coarse results, and then use neural renderer to get fine results. Only small driving part needs to be run repeatedly to drive the expression, while the rest is executed only once.
3.1 Dual-lifting and Reconstruction Branch
Given an input source image, our goal is to reconstruct detailed 3D head avatar. To ensure stable modeling and learning, we impose certain constraints on the reconstruction process. First, we assume that the reconstructed head always located the origin in normalized 3D space. Second, the rotation of the head is modeled through changes in camera pose ensure that the head itself is relatively stationary. We follow the same strategy when tracking 3DMM parameters and camera parameters from training and testing data. These constraints allow the model to effectively utilize the stable priors the human head topology.
Leveraging the success of 3D Gaussians splatting [Kerbl et al., 2023] in synthesis quality and rendering speed, we propose dual-lifting method to reconstruct 3D Gaussians from single image. Reconstructing 3D Gaussians typically requires millions of points, but obtaining such a dense density of Gaussian points from a single image is challenging task, especially without test-time optimization. To address this problem, we propose novel reconstruction method: the dual- method. Briefly, we first get the local feature plane Floest by frozen DINOv2 backbone, and then predict the offsets of each pixel relative to the feature plane and the other necessary parameters (including color, opacity, 3D Gaussians directly. We then map plane back to
scale and rotation), instead predicting 3D space based on the camera pose and place the plane through the origin, which provides the 3D position and normal vector of the plane pixels. Finally, we can calculate the position these 3D Gaussians in 3D space based on the predicted offiets, positions and normal vector. This process can be described as follows:
Gpos =pp.+ + Ecconno (Fiocal) nas P. Econnt (Flocat) ral.
(1)
Gc.o.e.r = [ECovoo(Focee
Econv1 Coonvi(Fiooull (Fiocut]]
(2)
where pt is the initial points plane mapped based on the estimated camera pose of 7, and passes through the origin. The size of p. is 296 x 296, which is consistent with the local feature Fioeat- Ecore0.1 are convolutional networks, n. is the normal vector of ps. Gpoe is the position of recon- structed 3D Gaussians, and Geo.s.r represents the color, opacity, scale, and rotation of 3D Gaussians.
It's worth noting that while predicting one set of lifting distances from the plane is possible, we adopted strategy of predicting forward and backward lifting separately. Our dual-lifting method aims to predict complete 3D structure from a single source image, to achieve multi-vier consistency during inference. If we predict only one set of lifting distances from the image plane, we may face some ambiguous situations during learning. For example, when we want to reconstruct a side view source image, predicting one set of lifting will simultaneously lift the point forward to the visible surface and backward to include the other side of the head. During this process, each pixel can be lifted to the visible surface to the opposite surface, as both are justified, resulting in model

---

performance degradation. Unlike single-lifting prediction, our dual-lifting strategy predicts forward and backward lifting separately, which eliminates ambiguities and stabilizes the optimization process.
Our dual-lifting method effectively exploits the detailed information of the source image to reconstruct 3D Gaussians. At the same time, the two sets of predicted lifting points can form an almost closed Gaussian points distribution, thus enhancing the performance of large viewpoint changes. The 3D Gaussian generated by dual-lifting can be rendered from any viewpoint, producing static results. In the next section, we describe how to control the facial expressions of the generated avatar.
3.2 Expression Branch
Expression transfer is not a straightforward task, but the 3DMM [Li et al., 2017] provides us with a powerful tool to represent common facial expressions and decouple expressions from identity, thereby facilitating expression control. Our expression branch establishes 3D Gaussians based on the 3DMM vertices to control the expressions of the generated images. To achieve this, bind learnable weights to each vertex in the 3DMM. Due to the stable semantics of 3DMM vertices, the features of these points correspond facial positions such as the eyes and mouth.
As shown in Fig. 2, given the source image I, and driving image Ia. we concatenate the global features Fa with the learnable features of vertices. We then use a MLP to predict the Gaussian parameters (excluding position) of each point from these features, and use the position of the 3DMM vertices. Here we combine the global features Fia of the source image when predicting the expression Gaussians. This will introduce identity information to the expression branch and enhance the identity consistency under various expressions, confirmed by our experiments. Throughout the driving process, we only need to infer the Gaussians of the reconstruction branch and expression branch once. Reenactment is achieved by modifying the camera pose and position of the Gaussians in the expression branch, which allows us to perform fast reenactment without redundant calculations.
3.3 Neural Renderer
Reconstructing 3D Gaussians typically requires millions of points, but in our dual- lifl-lifting method, we generate only 175,232 points. These Gaussians can reconstruct the target avatar, but with RGB information alone insufficient for capturing the rich details of a human avatar. To enhance the representation capability of the sparse Gaussians, we predict 32-dimensional features containing RGB information and then perform splatting to obtain coarse images. Then we use a popular neural renderer following existing methods [Li et al., 2023a, Chu et al., 2024, Ye et al., 2024] to get the fine image, as Fig. shows. Unlike these methods which use neural render as a super-resolution module to reduce rendering time, we do not upsample the image time issues. Our neural renderer effectively decodes the dual lifting and expression Gaussians features into RGB values, producing high-quality results and resolving potential conflicts between the two sets of Gaussians. We train our neural renderer from scratch during the training process, without any pre-trained initialization.
method do not face significant rendering
3.4 Training Strategy and Loss Functions
With the exception of the frozen DINOv2 backbone, we train the model from scratch. During training, we randomly sample two images from the same video, one as the source image and the other as the driving image and target image. Our primary objective during training is to ensure that the reenacted coarse and fine image aligns with the target image. Given that both images share the same identity, this alignment is achievable. We employ LI loss and perceptual loss [Johnson et al., 2016, Zhang et al., 2018, Ye et al., 2024] on both the coarse and the fine image.
Additionally, we propose a lifting distance loss Lisfting to assist dual-lifting learning. With the help of the prior provided by the tracked 3DMM, we require the lifting distance predicted by the network to be as close as possible to the 3DMM vertices. Specifically, we look for the lifting point closest to each 3DMM vertex and constrain their distance through L2 loss. The calculation as follows:
Luifting || admmm (argmingeeG...lll llp all p E Paámm. } I|:
where the Pydmm the tracked 3DMM vertices, Gpos the dual-lifting points, argmin find the nearest point. Our lifting distance loss leverages 3DMM priors. Additionally, since we constrain only
5

---

Source
Driving
GOHA
CVTHead
GPAvatar
Real3DPotrait
Potrait4D
Pvt
Ours
Figure 3: Cross-identie qualitative results on the VFHQ [Xie al., 2022] dataset. Compared with baseline methods, our method has accurate expressions and rich details.
a subset of dual-lifting points, model can still learn areas not modeled by 3DMM, such as hair and accessories. Experiments show Liitting can improve the 3D structure and the performance of large view changes.
The overall training objective is as follows:
L = 1/ 1ell + ,|I - Iell + (llv((L))) (++))l |1x(1,) ((+)ll) + DeLiittings
where I is target image, 1. and , are the generated coarse and fine image, Ap and A are the weights used balance the losses.
Experiments
4.1 Experiment Setting
Datasets. We use the VFHQ [Xie et al., 2022] dataset to train our model, which comprises clips from various interview scenarios. To avoid consecutive similar frames, we sampled 25 75 frames from the original video depending on video length. This resulted in dataset that includes 586.382 frames from 15,204 video clips. All the images are resized to 512x512. We tracked camera poses, FLAME [Li et al., 2017] parameters and removed the background following [Chu et al., 2024]. For evaluation, we use sampled frames from the VFHQ original test split, consisting of 5000 frames from 100 videos. The first frame of each video serves the source image, with the remaining frames used

---

Table 1:
Quantitative results on the VFHQ [Xie et al., 2022] dataset. We use colors to denote the first, second and third places respectively.
Self Reenactment I.PIPS CSIM
Cross Reenactment CSIM
Method
PSNR1
SSIM
AED]
APD]
AKD.,
AED APD]
StyleHeat YYin et al., 2022]
19.95
0.726
0.211
0.537
0.199
0.385
7.659
0.407
0.279
0.551
ROME [Khakhulin et al,, 2022]
19.96
0.786
0.192
0.701
0.138
0.186
4.986
0.530
0.259
0.277
OTAvatar [Ma ct al., 2023]
17.65
0.563
0.294
0.465
0.234
0.545
18.19
0.364
0.324
0.678
HideNcRF [I.i et al., 2023b]
19.79
0.768
0.180
0.787
0.143
0.361
7.254
0.514
0.277
0.527
GOHA [1.i et al., 2023a]
20.15
0.770
0..19
0.664
0.176
0.173
6.272
0,518
0.274
0.261
CVTHead [Ma et al., 2024a]
18.43
0.706
0.317
0.504
0.186
0.224
5.678
0.374
0.261
0.311
GPAvatar [Chu et al., 2024]
21.04
0.807
0.150
0.772
0.132
0.189
4.226
0,564
0.255
0.328
Real3DPortrait [Ye et al., 2024]
20,88
0.780
0.154
0.801
0.150
0.268
5.971
0.663
0.296
0.411
Portrait4D [Deng al., 2024b]
20.35
0.741
0.191
0.765
0.144
0.205
4.854
0.596
0.286
0.258
Portrait4D-v22 [Deng et al., 2024a]
21.34
0.791
0.144
0.803
0.117
0.187
3.749
0.656
0.268
0.273
Ours
21.83
0.818
0.122
0.816
0.111
0.135
3.349
0.633
0.253
0.247
Table 2: Quantitative results on the HDTF [Zhang et al., 2021] dataset. We use colors to denote the first, second and third places respectively.
Self Reenactment
Cross Reenactment CSIM AED! APD.
Method
PSNRt
SSIM
LPIPS
CSIM AED
APD
AKD.
StyleHeat [Yin et al., 2022]
21.41
0.785
0.155
0.657
0.158
0.162
4.585
0.632
0.271 0.239
ROME [Khakhulin et al., 2022]
20.51
0.803
0.145
0.738
0.133
0.123
4.763
0.726
0.268 0.191
OTAvatar [Ma et al., 2023]
20.52
0.696
0.166
0.662
0.180
0.170
8.295
0.643
0.292 0.222
HideNeRF [Li et al., 20236]
21.08
0.811
0.117
0.858
0.120
0.247
5.837
0.843
0.276 0.288
GOHA [Li et al., 2023a]
21.31
0.807
0.113
0.725
0.162
0.117
6.332
0.735
0.277 0.136
CVTHead [Ma et al., 2024a]
20.08
0.762
0.179
0.608
0.169
0.138
4.585
0,591
0.242 0.203
GPAvatar [Chu et al., 2024|
23.06
0.855
0.104
0.855
0.114
0.135
3.293
0.842
0.268
0.219
Real3DPortrait [Ye al., 2024]
22.82
0.835
0.103
0.851
0.138
0.137
4.640
0,903
0.299
0.238
Portrait4D [Deng al., 2024b]
20.81
0.786
0.137
0.810
0.134
0.131
4.151
0.793
0.291
0.240
Portrait4D-v2 [Deng et al., 2024a]
22.87
0.860
0.105
0.860
0.111
0.111
3.292
0.857
0.262
0.183
Ours
23.13
0.863
0.103
0.862
0.110
0.111
2.985
0.851
0.231
0.181
as driving and target images for reenactment. We also evaluate on HDTF [Zhang al., 2021] dataset, following the test split used in [Ma et al., 2023, Li et al., 2023a], including 19 video clips.
Implementation details. Our framework is built on the PyTorch [Paszke et al., 2017] platform. We use FLAME [Li et al., 2017] as our driving 3DMM. During training, we use the ADAM [Kingma and Ba, 2014] 4] optimizer with learning rate of 1.0e-4. The DINOv2 [Oquab et al., 2023] backbone is frozen during training and is not trained or fine-tune fine-tuned. Our training consists of 200,000 iterations with total batch size of 8. The training process is conducted on an NVIDIA Tesla A100 GPU and takes approximately 46 GPU hours, demonstrating efficient resource utilization. During inference, method achieves 67 FPS on an A100 GPU while using only 2.5 GB of VRAM, showcasing high efficiency. Further implementation details of the model can be found in the supplementary materials.
4.2 Main Results
Baseline methods. We conduct comparisons with existing state-of-the-art methods, including ROME [Khakhulin et al., 2022], StyleHeat [Yin et al., 2022], OTAvatar [Ma et al., 2023], HideN- cRF [Li et al., 2023b], GOHA [Li et al., 2023a], CVTHead [Ma et al., 2024a], GPAvatar [Chu et al., 2024], RealtDrrttratt [Ye et al., 2024], Portrait4D [Deng et al., 2024b], and Portrait4D-v2 [Deng et al., 2024a]. For each method, we use the official implementation to obtain the result. It is worth noting that actually the core contributions of Portrait4D-v2 are orthogonal to our work. They intro- duced new data generation method and novel learning paradigm to improve performance, which means our method can also benefit from their advancements.
Qualitative results. Fig. 3 shows qualitative comparisons between methods. Compared with other methods, our method can reconstruct detailed head avatars from source images and capture subtle facial movements such as eyes and mouth in driving images. Our method can also maintain identity
7

---

Source
Drive
Ours
One-plane
w/o Fin
w/o Rweuraz
w/o hrstiin
Figure 4:
Ablation results on VFHQ [Xie et al., 2022] datasets. We can see that our full method
performs best, especially on facial edges such glasses in large view angles.
consistency and image quality when handling large head rotations. At the same time, our method achieves high-quality reconstruction and rendering much faster speed than the baseline method.
Quantitative results. We also quantitatively evaluate the self and cross-identity reenactment perfor- mance between methods. For self-reen eef-reenactment with ground truth available, measure the quality of the synthesized images using PSNR, SSIM, LPIPS [Zhang et al., 2018] between the synthetic results and the ground truth. For identity similarity, caleulate the cosine distance of face recognition features [Deng et al., 2019a] between the reenactment results and the source images. For expression and pose, we use the average expression distance (AED) and average pose distance (APD) measured by a 3DMM estimator [Deng et al., 2019b], and the average keypoint distance (AKD) based on a facial landmark detector [Bulat and Tzimiropoulos, 2017] to evaluate the accuracy of driving control. For the cross-identity reenactment task, due to the lack of ground truth, we evaluate CSIM, AED, and APD, generally consistent with previous work [Li al., 2023a, Chu et al., 2024, Ye et al., 2024].
Tab. and Tab. show the quantitative results on the VFHQ and HDTF datasets, respectively. Our method outperforms previous methods in terms of reconstruction and synthesis quality and expression control accuracy but the cross-reenactment identity consistency slightly worse than some existing methods. We believe this is due the 3DMM [Li al., 2017] and 3DMM tracker we rely on, whose identity parameters and expression parameters are not completely decoupled. Some methods [Deng al., 2024b,a] that not based 3DMM have brought some inspiration to solve this limitation, and we leave these to future work. Importantly, our model not only achieves these quantitative results, but also achieves the real-time reenactment speed, which much faster than existing methods.
Inference speed and efficiency. Our method can run at 67 FPS on an A100 GPU with the naive Py framework and official 3D Gaussian Splatting implementation. As shown in Tab. 3, are the first real-time method for animatable one-shot head avatar reconstruction, which shows the application prospects and unique value of our method.
Table 3: The time of reenactment is measured in FPS. All results exclude the time for getting driving parameters that can be calculated in advance and are averaged over 100 frames.
StyleHeat ROME OTAvatar HideNeRF GOHA CVTHead GPAvatar Real3D P4D P4D-v2
Ours
Driving FPS
19.82
11.21
0.12
9.73
6.57
18.09
16.86
4.55
9.49
9.62
67.12
4.3 Ablation Studies
Dual-lifting, To validate the effectiveness our proposed dual-lifting method, we compare it against a baseline that lifts points from a single plane. This baseline requires the model to simultancously lift points forward and backward from the image plane, sometimes creating ambiguities. The results in Tab. 4 and Fig. 4 show that dual-lifting significantly enhances reconstruction quality. Moreover, since the lifting performed only once per identity and subsequent expression driving does not require recalculation, dual-lifting does not impact the performance during reenactment.

---

Table 4: Ablation results on the VFHQ [Xie et al., 2022] dataset.
Self Reenactment
Cross Reenactment
Method
PSNR
SSIM
LPIPS
CSIM
AED
APD
AKD
CSIM1
AED
APD
one-plane lifting
21.34
0.802
0.158
0.781
0.127
0.170
3.810
0.581
0.272
0.290
w/o Fia
21.13
0.807
0.155
0.774
0.125
0.155
3.722
0,537
0.270
0.272
w/o neural renderer
20.34
0.789
0.138
0.788
0.147
0.202
4.763
0.623
0.300
0.353
w/o Lirring
21.64
0.812
0.148
0.800
0.119
0.151
3.563
0.620
0.261
0.252
Ours
21.83
0.818
0.122
0.816
0.111
0.135
3.349
0.633
0.253
0.247
Source
Ours Full
Lurning
Single plane
Gurs
w/o Engeing
Single-pon laane--pane
Figure 5: Lifting results of an in-the-wild image, include the front view and the top view. Points filtered by Gaussian opacity. We color two parts of the dual-lifting separately, and the black points are the image plane. can be seen that the lifted 3D structure relatively flat without Lisfting.
Lifting distance loss. We evaluate the influence of the lifting distance loss Lisfting by removing during training. Without lifting distance loss, we observed performance degradation as shown in Tab. and Fig. 4. Compared with our full method, removing the point distance constraint will make it more difficult to reconstruct high-quality 3D structures, especially facial edges.
3D structure of dual-lifting. We further analyze and compare the 3D structure of dual-lifting. We show the visualization of filtered lifting points in Fig. 5. It can be seen that in the case of single-plane lifting or without Listing. the model can learn the correct 3D lifting even without any explicit 3D constraints. However, dual-lifting can produce 3D Gaussian points away from the input angle, and the 3D structure is also more reasonable rather than relatively flat.
Global feature in expression branch. We conduct an ablation study by removing the global identity features Fa from the expression branch. The results in Tab. 4 and Fig. 4 indicate that removing Fia decreases the identity similarity (CSIM) of the results and the reenactment quality. This demonstrates the importance of incorporating identity information in the expression branch.
Neural renderer. Due to the sparsity of our reconstructed Gaussians, we increased the output dimensions and introduced neural renderer to refine the coarse images and features. This process is similar to the super-resoluticn module in EG3D [Chan et al., 2022], but our neural renderer does not increase the resolution of the results. The results in Tab. 4 and Fig. 4 show the performance of coarse results without neural rendering. It can be observed that we can obtain reasonable results even using only sparse Gaussians, but the neural renderer significantly improves detail and expression.
Extreme inputs. We present more qualitative results with extreme inputs in Fig. 6. For extreme expressions or common occlusions such as sunglasses, our method shows good robustness. Our model can also work well with low-quality images and challenging lighting conditions, but the details of reconstructed avatars are inevitably affected. For example, avatars reconstructed from blurred images lack details, while those from images with challenging lighting conditions have fixed lighting, such shadows the nose. However, these features also demonstrate that method can faithfully restore details and handle various extreme cases.
Resolve conflicts of dual-lifting and expression Gaussians. Although we attempt to bring the two sets of Gaussians closer, there are inherent conflicts since one set static and the other is dynamic. We show some results with conflicts in Fig. 7. It can be seen that the RGB values conflict when there a significant expression difference between the dual-lifting Gaussians and expression Gaussians, but these conflicts are well resolved after neural rendering. We believe this is because our Gaussians have 32-D features that contain more information than RGB values. The neural rendering module can act filter to integrate the two point sets using these features and resolving possible conflicts.

---

Input
Drive and Result
Drive and Result
Input
Drive and Result
Drive and Result
Low Quality
Challenging Light Condition
Occlusion
Extreme Expression
The robustness of our model. Our method can produce reasonable results for low-quality images, challenging lighting conditions, significant occlusions, and extreme expressions.
Figure 6:
Input
Drive
GuussiastRRBB Only)
Result
Input
Drive
Guustans(RGB Only)
Resalt
Figure The case where two sets of Gaussians conflict, the conflict is resolved after neural rendering. We believe that neural rendering resolves the conflict through the 32D features carried by Gaussians.
Conclusion
In this paper, we proposed novel framework for one-shot head avatar reconstruction and real- time reenactment. The key innovation of our method is the dual-lifting approach for one-shot 3D) Gaussian reconstruction, which estimates the Gaussian parameters in single forward pass. We also propose 3DMM- expression control method and a loss function that uses 3DMM priors to constrain the lifting process. Our experiments demonstrate that our method outperforms state-of- the-art baselines in both the quality of head avatar reconstruction and reenactment accuracy, with significant improvements in rendering speed. We believe method has wide range of potential applications due to its strong generalization capabilities and real-time rendering speed.
Broader impacts. Although our method has great potential in various applications, it also poses the risk of misuse, such as generating fake videos and spreading false information. We strongly oppose such misuse and have proposed several measures to prevent it, as detailed in Sec. E. With proper and responsible use, we believe our method can offer significant benefits in a wide range of applications such video conferencing and entertainment industries.
Limitations and future work. Despite its strengths, our method has certain limitations. For example our model may generate less detail for unseen areas, and our 3DMM-based expression branch cannot control the areas not modeled by 3DMM, such as hair and tongue. These limitations highlight the possible improvements in future work to increase the performance and practicality of our method. In Sec. F. we provide a more detailed discussion our limitations and future work.
Acknowledgements
This work was partially supported by JST Moonshot R&D Grant Number JPMJPS2011. CREST Grant Number JPMICR2015 and Basic Research Grant (Super Al) of Institute for Al and Beyond of the University of Tokyo. In addition, this work was also partially supported by JST SPRING, Grant Number JPMJSP2108.
10

---

References
Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang. Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujuu Yang. Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In European Conference on Computer Vision (ECCV), 2022.
Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu. Pirenderer: Controllable portrait image generation via semantic neural rendering. In IEEE International Conference on Computer Vision (ICCV), 2021.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Zhiyuan Ma, Xiangyu Zhu, Guojun Qi, Zhen Lei, and Lei Zhang, Otavatar: One-shot talking face avatar with controllable tri-plane rendering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Xueting Li, Shalini Mello, Sifei Liu, Koki Nagano, Umar Iqbal, and Jan Kautz. Generalizable one-shot neural head avatar. Arxiv, 2023a.
Xuangeng Chu, Yu Li, Ailing Zeng. Tianyu Yang, Lijian Lin, Yunfei Liu, and Tatsuya Harada. GPAvatar: Generalizable and precise head avatar from image(s). In The Twelfth International Conference on Learning Representations, 2024.
Yu Deng, Duomin Wang, and Baoyuan Wang- Portrait4d-v2: Pseudo multi-view data creates better 4d head synthesizer. arXiv preprint arXiv:2403. 2024a.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42, 2023.
Yuelang Xu, Benwang Chen, Zhe Li. Hongwen Zhang, Lizhen Wang, Zerong Zheng, and Yebin Liu. Gaussian head avatar: Ultra high-fidelity head avatar via dynamic gaussians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning model of facial shape and expression from 4D scans. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), pages 194:1-194:17, 2017.
Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1 1125-112 1134, 2017.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10-8811, 2020.
Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9459- 90999948,444 2019.
Egor Burkov, Igor Pasechnik, Artur Grigorev, and Victor Lempitsky. Neural head reenactment with latent pose descriptors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13786- 13795, 2020.
Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose- controllable talking face generation by implicitly modularized audio-visual representation. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 4176-44 4176-4186, 2021.
11

---

Duomin Wang, Yu Deng, Zixin Yin, Heung- Yeung Shum, and Baoyuan Wang. Progressive disentan- gled representation learning for ane-grained controllable talking head synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 17979-17999 11999, 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pages 4401-4410, 2019.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019.
Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2663-2677 2671. 2022.
Ta-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3397-3406, 2022a.
Bowen Zhang, Chenyang Qi, Pan Zhang, Bo Zhang, HsiangTao Wu, Dong Chen, Qifeng Chen, Yong Wang, and Fang Wen. Metaportrait: dlentity-preserving talking head generation with fast personalized adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22096-22100 2023a.
Ting-C Ting-Chun Wang, Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10039-10049, 2021a.
Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8652-8661, 2023b.
Volker Blanz and Thomas Vetter. morphable model for the synthesis of 3d faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 1999.
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. In 2009 sixth IEEE international conference on advanced video and signal based surveillance, pages 296-301, 2009.
Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Luthi, Sandro Schönborn, and Thomas Vetter. Morphable face models-an open framework. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 75-82, 2018.
Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5784-5794, 2021.
Jaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang Hc, Tianshu Hu, Jingtuo Liu. Gang Zeng, and Jingdong Wang. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition. arXiv preprint arXiv:2211.12362 2022.
Sicheng Xu, Jiaolong Yang, Dong Chen, Fang Wen, Yu Deng, Yunde Jia, and Xin Tong. Deep 3d portrait from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7710-7720, 2020.
Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Egor Zakharov. Realistic one-shot mesh-based head avatars. In European Conference on Computer Vision (ECCV), 2022.
Weichuang Li, Longhao Zhang, Dong Wang, Bin Zhao, Zhigang Wang, Mulin Chen, Bang Zhang, Zhongjian Wang, Liefeng Bo, and Xuelong Li. One-shot high-fidelity talking- synthesis with deformable neural radiance field. IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
12

---

Wangbo Yu, Yanbo Fan, Yong Zhang. Xuan Wang, Fei Yin, Yunpeng Bai, Yan-Pei Cao, Ying Shan, Yang Wu, Zhongqian Sun, et al. Nofa: Nerf-based one-shot facial avatar reconstruction. ACM SIGGRAPH 2023 Conference Proceedings, pages 1-12, 2023.
Zhenhui Ye, Tianyun Zhong, Yi Ren, Jiaqi Yang, Weichuang Li. Jiawei Huang, Ziyue Jiang, Jinzheng He, Rongje Huang, Jinglin Liu, et al. Realdd-portratt: One-shot realistic 3d talking portrait synthesis. arXiv preprini arXiv:2401 1.85003, 2024.
Yu Deng, Duomin Wang, Xiaohang Ren, Xingyu Chen, and Baoyuan Wang. Portrait4d: Learning one-shot 4d head avatar synthesis using synthetic data. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024b.
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Scitz, and Ricardo Martin- Nerfies: Deformable neural radiance fields. IEEE International Conference Computer Vision (ICCV), 2021a.
Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael Black, and Otmar Hilliges. Pointavatar: Deformable point-based head avatars from videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21057-21067, 2023.
Yunpeng Bai, Yanbo Fan, Xuan Wang, Yong Zhang, Jingxiang Sun, Chun Yuan, and Ying Shan. High- fidelity facial avatar reconstruction from monocular video with generative priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023a.
Taekyung Ki, Dongchan Min, and Gyeongsu Chae. Learning to generate conditional tri-plane for 3d-aware expression controllable portrait animation. arXiv preprint arXiv:2404.00 000666, 2024.
Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8649-8666 8009-865,,222 2021.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In Proceedings of the IEEF/CVF International Conference on Computer Vision, 2021.
Yang Hong, Bo Peng, Haiyao Xiao, Ligang Liu, and Juyong Zhang. Headnerf: A real-time nerf-based parametric head model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 20374-20384, 2022b.
ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu. Rignerf: Fully controllable neural 3d portraits. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 2022.
Keunhong Park. Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan Goldman, and Steven M. Seitz. Hypemerf: A higher- nigher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph., 2021b.
Ricardo
Xuan Gao, Chenglai Zhong, Jun Xiang, Yang Hong, Yudong Guo, and Juyong Zhang. Reconstructing personalized semantic facial nerf models from monocular video. ACM Transactions on Graphics (TOG), 41, 2022.
Ziqian Bai, Feitong Tan, Zeng Huang. Kripasindhu Sarkar, Danhang Tang, Di Qiu, Abhimitra Meka, Ruofei Du, Mingsong Dou, Sergio Orts-Escolano, et al. Learning personalized high quality volumetric head avatars from monocular rgb videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023b.
Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, and Matthias Nießner. Nersemble: Multi-view radiance field reconstruction of human heads. ACM Transactions on Graphics (TOG), 42, 2023.
Xiaochen Zhao, Lizhen Wang, Jingxiang Sun, Hongwen Zhang, Jinli Suo, and Yebin Liu. Havatar: High-fidelity head avatar via facial model conditioned neural radiance field, ACM Transactions on Graphics, 43, 2023.
13

---

Zicheng Zhang. Ruobing Zheng, Ziwen Liu, Congying Han, Tianqi Li, Meng Wang, Tiande Guo, Jingdong Chen, Bonan Li, and Ming Yang. Learning dynamic tetrahedra for high nigh-quality talking head synthesis. arXiv preprint arXiv:2402. 00:400..77000 17364, 2024.
Zhongcong Xu, Jianfeng Zhang, Junhao Liew, Wenqing Zhang, Song Bai, Jiashi Feng, and Mike Zheng Shou. Pv3d: A 3d generative model for portrait video generation. In The Tenth International Conference Learning Reprexentations, 2023a.
Junshu Tang, Bo Zhang, Binxin Yang, Ting Zhang, Dong Chen, Lizhuang Ma, and Fang Wen. 3dfaceshop: Explicitly controllable 3d-aware portrait generation. JEEE Transactions on Visualiza- fion & Computer Graphics, 2023.
Kegiang Sun, Shangzhe Wu, Zhaoyang Huang, Ning Zhang, Quan Wang, and HongSheng Li. Controllable 3d face synthesis with conditional generative occupancy fields. Advances in Neural Information Processing Systems, 35, 2022.
Hongyi Xu, Guoxian Song. Zihang Jiang, Jianfeng Zhang, Yichun Shi, Jing Liu, Wanchun Ma. Jiashi Feng, and Linjie Luo. Omniavatar: Geometry-guider controllable 3d head synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12814-12828 2023b.
Peiye Zhuang, Liqian Ma, Sanmi Koyejo, and Alexander Schwing. Controllable radiance fields for dynamic face synthesis. In 2022 International Conference on 3D Vision (3DV), 2022a.
Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, and Yebin Liu. Next3d: Generative neural texture rasterization for 3d-aware head avatars. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, 2023.
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Transactions on graphics (TOG), 2022.
Jiaxin Xie, Hao Ouyang, Jingtan Piao, Chenyang Lei, and Qifeng Chen. High-fidelity 3d gan inversion by pseudo-multi-vin optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 321-331, 2023.
Yiyu Zhuang, Hao Zhu, Xusen Sun, and Xun Cao. Mofanerf: Morphable facial neural radiance field. In European conference on computer vision, 2022b.
Haoyu Ma, Tong Zhang, Shanlin Sun, Xiangyi Yan, Kun Han, and Xiaohui Xie. Cvthead: One-shot controllable head avatar with vertex-feature transformer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024a.
Songlin Yang. Wei Wang. Yushi Lan, Xiangyu Fan, Bo Peng, Lei Yang, and Jing Dong. Learning dense correspondence for nerf-based face reenactment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024.
Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Talkinggaus- sian: Structure Structure-persistent persistent 3d talking head synthesis via gaussian splatting. arXiv preprint arXiv:2404. 152644 2024.
Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, and Liqiang Nie. Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians. arXiv preprini arX1v:2312.02/134, 2023.
Cong Wang, Di Kang, He-Yi Sun, Shen-Han Qian, Zi-Xuan Wang, Linchao Bao, and Song- Zhang. Mega: Hybrid mesh- gaussian head avatar for high-fidelity rendering and head editing. arXiv preprint arXiv:2404. 19026, 2024a.
Shengjie Ma, Yanlin Weng, Tianjia Shao, and Kun Zhou. 3d gaussian blendshapes for head avatar animation. arXiv preprint arXiv:2404. 19398, 2024b.
Shengze Wang, Xueting Li, Chao Liu, Matthew Chan, Michael Stengel, Josef Spjut, Henry Fuchs, Shalini De Mello, and Koki Nagano. Coherent 3d portrait video reconstruction via triplane fusion. 2024b.
arXiv preprint ar
14

---

Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2023.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po- Yao Huang, Hu Xu, Vasu Sharma, Shang-Wer Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1114, 2016, Proceedings, Part II 14. Springer, 2016.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and benchmark for video face super-resolution In The IEEE Conference Computer Vision and Pattern Recognition Workshops (CVPRW), 2022.
Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face generation with high-resolution audio-visual dataset. In Proceedings the IEEE/CVP Conference on Computer Vision and Pattern Recognition, 2021.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 44/2.6980 2014.
Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019a.
Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image image set. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 2019b.
Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In Proceedings of the IEEE international conference on computer vision, 2017.
Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021b.
Matthew Tancik, Ben Mildenhall, and Ren Ng. Stegastamp: Invisible hyperlinks in physical pho- tographs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 2117-2126, 2020.
15

---

A Reproducibility
A.1 More Implementation Details
Specifically, we use DINOv2 Base as our feature extractor, which takes 3 x 518 x 518 images as input, and encodes 296 X26 local feature maps and 768-dimensional global features. We then obtain the Gaussian parameters of each pixel through 4 groups of ResBlocks He et al. [2016] without down- sampling. The dimension of Gaussian parameters is 41 dimensions, including 32 dimensions of color information, dimension of opacity information, 3 dimensions of scale information, 4 dimensions of rotation information, and dimension of lifting distance information. Since FLAME [Li et al., 2017] contains 5023 points, we assign a 256-dimensional feature to each point, so the total point feature size is 5023 256. We concatenate these features with global features to predict expression Gaussian parameters using an MLP with 1024 input dimensions. This MLP contains 6 layers, and since it does not include lifting distance, the output is 40 dimensions. Our neural renderer employs StyleUNet [Wang et al., 2021b] to map images from 32 x 512 x 512 to 3 x 512 x 512 dimensions. We also provide the code for the model in the supplementary material for reference.
A.2 More Data Processing Details
We use 15,204 video clips from the VFHQ dataset [Xie et al., 2022] for training and 100 videos for testing, following the original VFHQ split. For training videos, we uniformly sample frames based on the video's length: 25 frames the video is less than 2 seconds, 50 frames the video is to seconds, and 75 frames if the video is longer than 3 seconds. For testing videos, we uniformly sample 50 frames from each clip, resulting in a total of frames for training and 5,000 frames for testing. For the HDTF dataset, we use the test split from OTAvatar [Ma et al., 2023], which includes 19 videos. We uniformly sample 100 frames from each video, creating a test set with 1,900 frames.
For all these frames, we remove the background and resize them to 512 x 512 pixels. We extract and refine the 3DMM parameters for each frame following [Chu et al., 2024]. Although the labels generated by this automatic annotation method are somewhat noisy and imperfect, this approach allows us build large dataset, effectively mitigating the impact of data inaccuracies.
A.3 More Evaluation Details
We conduct comparisons with several state-of-the-arr methods, including ROME [Khakhulin et al., 2022], StyleHeat [Yin et al., 2022], OTAvatar [Ma et al., 2023]. HideNeRF [Li et al., 2023b], GOHA [Li et al., 2023a], CVTHead [Ma et al., 2024a]. GPAvatar [Chu et al., 2024], RealtDPortrait [Ye et al., 2024], Portrait4D [Deng al., 2024b], and Portrait4D-v2 [Deng et al., 2024a]. For each method, we use the official data pre-proces pre-processing script to process its input frame and driver frame, and use the official implementation to obtain the result frame. To ensure fair comparison, we realign the results from all methods, as some methods crop and center the face region while others do not. Specifically, detect landmarks and crop the head region at the same size for all driving images and results, and then resize the results to 512x512 for evaluation.
It is worth noting that although Portrait4D and Portrait4D-v2 achieve the same functionality and get really good results, their core contributions are orthogonal to our work. They introduce a new data generation method and new learning paradigm, which means our method can also benefit from their advancements. We leave the integration of these parallel works to future research.
B Preliminaries of 3DMM
We utilize a widely-used 3D morphable model (3DMM): the FLAME [Li et al., 2017] model which renowned for its geometric accuracy and versatility. This model is popular in applications such as facial animation, avatar creation, and facial recognition due to its realistic rendering capabilities and flexibility. We use it to work as our expression driven signal and geometry prior. The FLAME model represents the head shape as follows:
TP 0, v) = T + (S(3 + BP(Ū; P) BE(,; E),
(5)
where T is the template head avatar mesh (S S) is the shape blend-shape function to account for identity-related shape variation, ( P) is jaw and neck pose part to correct pose deformations
16

---

Reenactment and multi-view results of our method on images. From left to right: input image, driving image, driving and novel view results.
Figure 8:
that cannot be explained solely by linear blend skinning, and expression blend-shapes BE( E) used to capture facial expressions such closing eyes smiling.
C Per-part Rendering and 3D Lifting of Our Method
We present the results of rendering the dual-lifting Gaussians from the reconstruction branch and the Gaussian from the expression branch separately. As Fig, 9 shows, the dual-lifting Gaussians
17

---

Source
Driving
Ours
Dual-plane 1st Dual-plane 2st
Expression
Figure 9: Per-par rendering of the dual-lifting and expression Gaussians. We can see that the dual-lifting Gaussians reconstruct the head's base structure and facial details respectively. It worth noting that our Gaussians are not purely RGB Gaussians. Instead, our Gaussians include 32-D) features (as described in Sec. 3.3). We visualize the first dimensions of these features (i.e., the RGB values of the Gaussians) here without the neural rendering module. So this visualization intended intuitively display the functionality of each part and the importance of each branch should be judged based on RGB values alone.
Dual-ppans Lifting
bual plane
Duullppläne Lifting
Source
Source
Source
Lifting
Dual-lifting results of in-the-wild images. We that the dual-lifting point cloud has rich details, including glasses and hair. We color the two parts the dual-lifting separately, and the black points are the image plane.
Figure 10:
reconstruct the head's base structure and facial details respectively, which is in line with expecta- tions. We also show more lifted points Fig. 10, we can see that the dual-lifting point cloud has rich details, including glasses and hair. Additionally, we provide some lifting point cloud files in supplementary materials.
D More Qualitative Results
We show more self-identity qualitative comparisons with baseline methods in Fig. 11, and cross- identity qualitative comparisons Fig. 13. Here we show the results of all baseline methods on the VFHQ [Xie et al., 2022] dataset and HDTF [Zhang et al., 2021] dataset.
We also show more results of method and baseline methods for self and cross-identity reenactment. In Fig. 12, we not only show the reenactment results but also the multi-view results our method. In Fig. 16, we show more comparisons and consecutive frames and highlight the regions of interest. We also show more in-the-wild results of our method in Fig. 8, 14 and 15. It can seen that our method maintains good identity consistency and 3D consistency when the viewing angle changes.
Additionally, we provide a supplementary video to demonstrate video driving results. Although no special processing is performed, our method has timing-stable results on video generation.
18

---

Source
Driving
StyleHeat
ROME
OTAvatarHiideNekt GOHA
CVTHead GPAvatar
Real3D
P4D
P4D-v2
Ours
Figure 1I: Self-identity reenactment results on VTHQ [Xie et al., 2022] and HDTF [Zhang et al., 2021] datasets. The top six rows are from VFHQ and the bottom three rows are from HDTE
Source
Driving
Reenactment
Novel Views
Source
Driving
Reenactment
Novel Views
5
S
Figure 12: Reenactment and multi-vier results of our method on the VFHQ [Xie et al., 2022] dataset.
Our method can maintain consistency across multiple views.
19

---

Source
Driving StyleHeat ROME OTAvatarflideNeRFFOHA CVTHeadGPAvatar Real3D
P4D
P4D.v2
Ours
T
T
Figure 13: Cross-identity reenactment results on VFHQ [Xie et al., 2022] and HDTF [Zhang et al., 2021] datasets. The top ten rows are from VFHQ and the bottom four rows are from HDTF.
E More In-Depth Ethical Discussion
Our framework offers many applications but also presents ethical risks, such as the potential creation of fake videos ("deepfakes"), violations of privacy, and the dissemination of false information. We do not advocate such misuse and have proposed several measures to prevent these risks:
Watermarking technologies. To ensure transparency and prevent misuse, we plan to employ watermarking techniques in code that will be released. Visible watermarks enable viewers to immediately recognize content as Al-generated, helping them distinguish potential misuse. In addition to visible watermarks, we plan to embed invisible watermarks [Tancik et al., 2020], which difficult to remove. These invisible marks help track and identify the source of videos, even if they are re-edited. This tracking capability encourages producers to consider the ethical implications and potential risks of their creations by storing information about the video producer.
Strict licenses. We will release our code and model under a striet license. The license will prohibit the synthesis of real individuals without explicit consent for commercial use. This ensures that our
20

---

Reenactment and nie results of our method on in-the-wild images. From left to right: input image, driving image, driving and novel view results.
Figure 14:
21

---

2
China PrOSCLATOn
Figure 15:
Reenactment and multi-view results of our method on in-the-wild images. From left to right: input image, driving image, driving and novel view results.
22

---

Source
Driving
GOHA
CVTHead
GPAvatar
Reat3DPotrait
Potrait4D
Potrain4Dv2
Ours
Input
Drive
GPAvatar
Real3D
P4Dv2
Ours
Drive
GPAvatar
Real3D
P4Dv2
Ours
Figure 16:
Qualitative results and video continuous frame results with highlighted attention regions. selected competitive methods to show continuous frames. Better to view zoomed
Input
Drive
Result
Input
Drive
Result
Figure 17: Our model has some limitations. For example, the tongue is not modeled and the unseen regions of the input have less details. Better view it zoomed in.
technology is used ethically and prevents it from violating the consent of the individual represented by the avatar. Illegal misuse can be traced through the watermark system.
In summary, we will implement robust safeguards to prevent the misuse our head avatar recon- struction system. We urge video creators to be mindful of the ethical responsibilities and potential risks when using talking face generation technologies. With careful and responsible use, our method provide substantial benefits across various real-world applications.
F Limitations and Future Work
Although our method achieves high-quality synthesis results compared to previous approaches, there still some limitations. When rendering synthesized results from novel views, unseen areas in the original source image often lack detail and may produce results with statistically average expectations. For example, generating the other half of the face from side view input generating an open mouth from an input image with a closed mouth. Additionally, our expression branch is based on 3DMM and learned from VFHQ video data. This branch may not capture extreme facial movements or parts not modeled by 3DMM, such as one eye being open and the other being closed, the tongue, and hair. We show the qualitative results of these limitations Fig. 17. Future work may involve learning expression embeddings [Deng et al., 2024b] directly from images, alleviating data requirements and tracking accuracy needs through data generation [Deng et al., 2024a], gathering more expressive data to improve expression imitation. Extending our approach to handle full-body avatar synthesis is also a promising direction for future research.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction include the claims made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We describe in the limitations and future work section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We describe in the implementation details section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification: We have included the core code in the supplementary material and will release the full code after refactoring and cleanup. Our data uses publicly available data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.c cc/
httpe/guidee//eeeeeeeeeee wwtpuuubissssoooooooioooooo conooccc)))) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. cttpcc/ppbbc. mubbic/guiies. /CodeSubmisss ietmiissionnl swsssssoiccyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We describe in the implementation details section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We use fixed random seeds in all experiments to ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe in the implementation details section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eerrssceeeeee de cswuidellllllliie ines?
Answer: [Yes]
Justification:
We fully follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss it in broader impacts section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discuss it in the broader impacts section.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: It is included in the source code.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We not ready release model yet.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "S55kgMyE4t", "reviewer": "Reviewer_PdVU", "summary": "This paper presents \"Generalizable and Animatable Gaussian Head Avatar\" (GAGA), a method for one-shot animatable head avatar reconstruction using 3D Gaussians. Unlike existing methods that depend on neural radiance fields and require extensive rendering time, GAGA employs a dual-lifting method to generate high-fidelity 3D Gaussians from a single image in a single forward pass. The approach integrates global image features and 3D morphable models to control expressions and achieve real-time reenactment speeds. Experiments demonstrate that GAGA outperforms previous methods in reconstruction quality and expression accuracy.", "questions": "1. How sensitive is the model to variations in the input image quality and lighting conditions?\n2. How robust is the model when applied to images with significant occlusions or extreme facial expressions?\n3. What specific measures were taken to ensure the diversity and quality of the training data?", "limitations": "1. The model may produce less detail for unseen areas, which can affect the realism of the generated avatars in dynamic scenes.\n2. The expression control is limited by the constraints of the 3DMM, which does not model elements like hair and tongue, leading to potential inaccuracies.\n3. There is a slight compromise in identity consistency of cross-identity reenactment due to the partial decoupling of identity and expression parameters in the 3DMM.\n4. While the method achieves real-time speeds, the dual-lifting and neural rendering processes add complexity to the implementation and may require significant computational resources for training.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The dual-lifting method for reconstructing 3D Gaussians from a single image is novel and effective, allowing for high-fidelity reconstruction.\n2. Efficient resource utilization during training and inference makes it practical for real-time applications, which is a significant improvement over existing methods that are often slow.\n3. The model can generalize to unseen identities without specific optimizations, broadening its applicability.\n4. Experimental results show superior reconstruction quality and expression accuracy performance compared to state-of-the-art methods.", "weaknesses": "1. The reliance on 3D morphable models for expression control may limit the model's flexibility.\n2. The 3DMM identity and expression parameters are not entirely decoupled, potentially impacting cross-identity reenactment identity consistency.\n3. The method may generate less detail for unseen areas and has limitations in controlling regions not modeled by 3DMM, such as hair and tongue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and valuable comments. We are pleased to address your concerns in the following sections:\n\n**How sensitive is the model to image quality and lighting conditions?**\n\n* We present more qualitative results with low-quality images or challenging lighting conditions in Fig. 1 of the new supplementary page.\nThe reconstructed avatars are inevitably affected by image quality and lighting conditions. For example, avatars reconstructed from blurred images lack details, while those from images with challenging lighting conditions have a fixed lighting condition, such as some shadows on the nose. However, these features also demonstrate that our model can faithfully restore details from the input image and handle scenarios with varying image quality and challenging lighting.\n\n**How robust is the model to significant occlusions or extreme facial expressions?**\n\n* In Fig. 1, we also show the results for inputs with significant occlusions and extreme expressions. For common occlusions such as sunglasses, the model can handle them well. For some uncommon occlusions such as hands, the reconstructed avatar will be affected by the occlusion. For extreme inputs or driving expressions, our method shows good robustness. This shows that our method can produce reasonable results even in extreme cases.\n\n**What specific measures were taken to ensure the diversity and quality of the training data?**\n\n* We constructed our training data from the VFHQ dataset, a talking head video dataset containing 15,204 video clips. Despite the repeated identities, the dataset still has a large number of unique identities, ensuring sufficient identity diversity in our training data.\nTo ensure diversity in expressions and head poses between frames, we extracted frames from the videos. As described in Sections 4.1 and A.1, we uniformly sampled 25-75 frames from each video based on its length. This temporally sparse sampling ensures that different frames exhibit as much variation in expressions and poses as possible.\n\n* For the ground truth 3DMM parameters in our dataset, we adopted the implementation from GPAvatar [Chu et al., 2024]. This implementation integrates state-of-the-art 3DMM estimators (MICA, EMOCA), resulting in high-quality 3DMM labels.\n\n**Implementation complexity and computational resources for training.**\n\n* Our method is also highly efficient in training. As shown in the table below, our training costs are lower than those of the baseline methods. In addition, as a generalizable method, our model only needs to be trained once and does not require fine-tuning during inference.\nIn the future, we plan to open-source our code, including implementations of all components, as well as our pre-trained models. We hope this will provide an easy-to-use and solid baseline for future research.\n\n* | Method |StyleHeat | ROME | OTAvatar | HideNeRF | GOHA | CVTHead | GPAvatar | Real3D | P4D | P4D-v2 | Ours|\n| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| **Time(GPU Hours)**  |166           |    -     | 192           | 1667           | 768      | 1200        | 50           | 1424     | 672  | 1536      |  **46** |\n\n* Among them, ROME did not provide training details and code. StyleHeat, HideNeRF, and CVTHead did not release official training time, so we conducted limited training and estimated the total time for these methods. All times tested on / reported on /or converted to A100.\n\n**About the limitations of 3DMM.**\n* We further discussed the impact of 3DMM in our rebuttal to reviewer **65B2**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YmWMyjd7z8", "reviewer": "Reviewer_3QbB", "summary": "The paper introduces a novel framework, GAGA, for one-shot animatable head avatar reconstruction from a single image. The key innovation is the dual-lifting method, which generates high-fidelity 3D Gaussians that capture identity and facial details by predicting lifting distances from the image plane to 3D space. This method leverages global image features and a 3D morphable model (3DMM) to ensure accurate expression control and real-time reenactment speeds. The model can generalize to unseen identities without specific optimizations. Experimental results demonstrate that GAGA outperforms previous methods in terms of reconstruction quality and expression accuracy. The main contributions include the introduction of the dual-lifting method, the use of 3DMM priors to constrain the lifting process, and the combination of 3DMM priors with 3D Gaussians for efficient expression transfer, allowing high-quality real-time rendering.", "questions": "- Why not also predict position offsets for the expression Gaussians? To keep expression structures, regularization can be used.\n- Would the two sets of Gaussians (expression & dual-lifted) contract each other during expression?\n- Why not bind the lifted Gaussians to the 3DMM vertices and animate together?", "limitations": "The paper acknowledges several limitations. First, the model may generate less detailed reconstructions for unseen areas of the face, such as the back of the head or the interior of the mouth, which are not visible in the input image. Second, the 3DMM-based expression branch cannot control parts of the head that are not modeled by 3DMM, such as hair and the tongue.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed dual-lifting of approach is interesting and enables the hallucination of unseen areas and resolves ambiguity of front/back 3D Gaussians.\n- The combination of expression Gaussians and dual-lifted Gaussians allows flexible expression control.\n- The method outperforms various baselines in a wide range of metrics. The rendering speed also significantly outperforms prior methods.\n- The paper is clearly written and easy to follow.", "weaknesses": "- The Gaussians from the expression branch only use the vertices of the 3DMM model, which can be inaccurate and unable to model fine facial features.\n- It would be nice to show more side-by-side video comparison with baselines. Many baselines look similar qualitatively and it’s hard to tell the improvement without highlighting the difference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and insightful comments. We are happy to address your questions in the following:\n\n**Why not also predict position offsets for the expression Gaussians?**\n* Our method emphasizes the efficiency of inference rendering. Predicting expression Gaussians offset for each frame will impact the inference efficiency. Our expression Gaussians are controlled by 3DMM vertices now, which can be performed asynchronously with rendering and is very efficient. \n* In addition, the offset of the point is also related to the shape and expression. These two parameters determine the position of the point before the offset. At the same time, the number of points is large (5023), which makes it difficult to directly learn the offset of each expression point.\n* But we also experimented with adding additional offsets to the expression Gaussians. Our setup is as follows: we extract global features from the driving image, then use the initial positions given by the 3DMM and trainable point features to predict point offsets, and we apply regularization to prevent excessive displacement. The results are shown below.\n* | Method | PSNR↑ | SSIM↑ | LPIPS↓ | CSIM↑ | AED↓ | APD↓ | AKD↓ | CSIM↑ | AED↓ | APD↓ |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| with offsets | **21.93** | 0.814 | 0.138 | 0.797 | 0.121 | 0.168 | 3.732 | 0.584 | 0.263 | 0.294 |\n| Ours | 21.83 | **0.818** | **0.122** | **0.816** | **0.111** | **0.135** | **3.349** | **0.633** | **0.253** | **0.247** |\n* It can be seen that in such a setting, adding offset predictions has little effect or even worse impact on most metrics.\n\n**Would the two sets of Gaussians contract each other during expression?**\n* During the expression (inference) process, the two sets of Gaussians do not contract towards each other. But during training, we use MSE loss to bring the two sets of Gaussians closer together to fully leverage the priors of the 3DMM. So the two sets of Gaussians are close to each other after training.\n\n**Why not bind the lifted Gaussians to the 3DMM vertices and animate them together?**\n* Our primary consideration is efficiency. In our setup, we obtain 175,232 lifted Gaussians and 5,023 expression Gaussians, which require a large matrix to compute their binding relationship. Unlike full-body reconstruction, which involves significant deformations (requiring binding points to a few bones), facial reconstruction doesn't demand such extensive deformations. Therefore, we first attempted to merge the expression Gaussians and enhancement Gaussians directly without binding. The results indicate that this approach is effective for capturing expressions and jaw movements.\n* We have further discussed how we resolved the conflict between the lifting Gaussians and the expression Gaussians caused by not binding in our response to reviewer d3t7.\n\n**Why do we only use the vertices of the 3DMM model?**\n* The vertices are naturally compatible with the positions of the Gaussians in 3DGS. Additionally, 3DMM (FLAME) has 5,023 points, providing sufficient information for head expression control. To integrate 3DMM more effectively into 3DGS, we chose not to use edge and face information.\n* Moreover, the points in 3DMM contain richer information beyond just shape and expression parameters. Therefore, we opted not to redundantly input the shape and expression parameters of 3DMM into the model.\n\n**Side-by-side video comparison with baselines.**\n* Due to rebuttal restrictions, we are unable to provide additional videos for rebuttal. Instead, we have included some consecutive frames in the new supplementary page and highlighted the areas of interest. We will add side-by-side video comparisons in our future open-source code and demo website.\n\n**About the limitations of 3DMM.**\n* We further discussed the impact of 3DMM in our rebuttal to reviewer **65B2**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6y2RJC3fJs", "reviewer": "Reviewer_65B2", "summary": "The paper proposes a method to achieve one-shot head avatar animation with 3D Gaussian Splatting (3DGS). With 3DGS, the papers show high-fidelity animation with fast inference speed. To solve one-shot 3DGS reconstruction, the paper proposes a dual-lifting method with 3DMM regularization. Experiments justify the model designs and show the proposed approach achieves SOTA performance.", "questions": "1. The current ablation study may not be sufficient to validate the dual-plane lifting. Is it possible to conduct a study by removing the reconstruction branch to better demonstrate the importance of dual-plane lifting?", "limitations": "Yes, the authors discuss their \"Limitations\" and provide an \"In-Depth Ethical Discussion\" proposing several measures to prevent these ethical risks.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a technically sound approach for solving the one-shot animatable head avatar problem using 3D Gaussian Splatting (3DGS), fully utilizing the pre-trained 2D network to obtain cues for lifting.\n2. The quantitative results are promising, and the qualitative results are sufficient to demonstrate the robustness of the approach.", "weaknesses": "1. The visual results of the method seem to be overly smoothed to some extent, making the results less realistic compared to some baselines (e.g., Portrait4Dv2). The authors may need to provide additional explanation for this and discuss why the method achieves better qualitative results than other methods.\n2. The approach is highly dependent on 3DMM but lacks a discussion about its impact. What if the estimation fails? How does the estimation accuracy affect the final performance (both training and inference)? If using other designs without 3DMM, what would be the method's performance?\n3. There should be some failure cases to show the limitations of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and helpful comments. We would like to address your concerns in the following sections:\n\n**Is it possible to conduct a study by removing the reconstruction branch to better demonstrate the importance of dual-plane lifting?**\n\n* Removing the reconstruction branch is possible, but it would result in a lack of detail. Our method integrates global identity features when predicting the expression Gaussians, which provides some identity information to the expression Gaussians. However, the 1D global features are insufficient to reconstruct detailed identity features. We provide qualitative results of the reconstruction branch in Fig. 2 on the new supplementary page and quantitative results here. The qualitative results show a severe lack of details on identity, demonstrating the necessity of the reconstruction branch, and the quantitative results also support this conclusion (the CSIM metric).\n* | Method | PSNR↑ | SSIM↑ | LPIPS↓ | CSIM↑ | AED↓ | APD↓ | AKD↓ | CSIM↑ | AED↓ | APD↓ |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| w/o Recons | 18.006 | 0.756 | 0.261 | 0.454 | 0.203 | 0.223 | 5.324 | 0.230 | 0.246 | 0.279 |\n| Ours (full) | **21.83** | **0.818** | **0.122** | **0.816** | **0.111** | **0.135** | **3.349** | **0.633** | **0.253** | **0.247** |\n\n**The visual results of the method seem to be overly smoothed, making the results less realistic compared to some baselines. The authors may need to provide additional explanation for this and discuss why the method achieves better qualitative results than other methods.**\n* Currently used quantitative metrics focus more on the correctness of results rather than their realism. For example, AED measures whether subtle expressions are correctly modeled, and PSNR and SSIM measure if identity details are authentically reproduced. Sometimes visually realistic results may be incorrect. To verify the realism of our results, we have provided our quantitative evaluation with FID scores (self-reenactment on VFHQ), which is a commonly used metric for realism. Our model demonstrates competitive performance compared to state-of-the-art methods.\n* | Method | StyleHeat | ROME | OTAvatar | HideNeRF | GOHA | CVTHead | GPAvatar | Real3D | P4D | P4D-v2 | Ours |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| **FID↓** | 72.138 | 49.516 | 70.692 | 51.930 | 39.638 |109.054 | 37.610 | 38.999 | 46.965 | 30.573 | **28.938** |\n\n* Our method tends to over-smooth the hair regions. We believe this because our method does not include control of these regions. During training, the hair and upper body will always look like the input image instead of the target image, and this inconsistency leads to over-smoothing. This problem is also observed in other baseline methods since all the baseline methods do not support hair control. Among them, GOHA, Real3DPotrait, and Potrait4D use GAN loss to enhance the realism, but lead to inaccurate expression control and false illusion of details, while Portrait4DV2 improves hair realism by using data generation to obtain correct ground truth of uncontrolled regions. But it's worth noting that the primary contribution of Portrait4DV2 is a new learning paradigm, which is orthogonal to our main contribution. Our method can also benefit from the data generation method of Portrait4DV2.\n\n**The approach is highly dependent on 3DMM but lacks a discussion about its impact.**\n* Thank you for pointing out some missing parts of the paper. We will include this discussion on 3DMM in the revised version.\n* 3DMM estimation is important in our method. We use the 3DMM estimation methods provided by GPAvatar (based on EMOCA and MICA) to process our training and inference data. While these methods introduce some errors during our training process, their robustness ensures that our model can still be effectively trained. In the future, our method will also benefit from advancements in 3DMM estimation; the more accurate the estimation, the better our model is expected to perform.\n* During inference, if the 3DMM model fails to accurately predict the target expression from the target image, our model will also reproduce these inaccuracies in the driving result. For example, some subtle expressions are not captured by 3DMM (subtle frowns), or the expression is incorrect due to the non-decoupling of identity and expression. Mitigating this issue depends on further developments in the 3DMM estimation field.\n* Control without the 3DMM is possible but would require significant modifications to the method. We hope to leave this for future work.\n\n**There should be some failure cases to show the limitations of the work.**\n* We show some limitations in Fig. 3 on the new supplementary page. This includes the results of the tongue that are not modeled by the 3DMM, and the model's lack of details in invisible areas in the input image. These examples will also be integrated into future revisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "192oTeaS5i", "reviewer": "Reviewer_d3t7", "summary": "This paper presents a method for animatable facial avatar reconstruction from a single RGB image, and the reconstructed avatar is based on 3D Gaussian splatting (3DGS) to support real-time rendering. To this end, the proposed method generates pixel-aligned Gaussian point clouds to reconstruct the identity, and use 3DMM to construct an additional set of Gaussian points to control the facial expression. The network is trained on a large scale corpus of human talking videos. After training, the network can produce reenactment results at real-time framerate for any unseen identities. Experiments show that the proposed method is able to produce plausible reenactment results.", "questions": "One minor suggestion: it would be better if the authors could provide some qualitative comparison against state-of-the-art methods in the form of dynamic reenactment videos and highlight the difference/advantages of the results.", "limitations": "The authors have discussed the limitation and potential social impact in Sec.5 of the main paper and Sec.E & F of the supplemental document.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* This paper proposes the first generalizable 3D Gaussian-based avatar modeling method (if not taking ECCV 2024 papers into account). \n\n* The authors compare the proposed method with most recent state-of-the-art baselines (such as Protrait-v2). The comparison is comprehensive and convincing.  \n\n* Results in Table 1 and Table 2 show that the proposed method consistently outperforms existing approaches under differently metrics and different settings.", "weaknesses": "* My major concern is about the method design. Despite the plausible results, the proposed method is confusing and doesn't make sense to me. If I understand correctly, the reconstruction branch produces Gaussian points that are aligned with the input image, and these points keep static and unchanged when performing reenactment. Instead, the expressions are modeled through the additional Gaussian points that are attached to the 3DMM surface (Line 187-188), which means these expression Gaussian points should play a major role in expression control. However, according to Figure 7, the Gaussian points produced by the reconstruction branch actually play a major role in facial rendering, while the \"expression Gaussians\" seem to be much less important. That doesn't make sense to me: how could the static points produced by the reconstruct branch model the dynamic expressions? When overlapping the reconstruction Gaussian points with the expression ones, how does the proposed method resolve the potential conflicts of these two point sets, such as different lip positions?\n\n\n* The so-called dual-lifting technique looks straight-forward and trivial to me. According to Sec.3.1, this module produces two pixel-aligned Gaussian point sets, one for the visible surface and the other for the invisible surface. However, using two set of points (visible+invisible, or front+back) to model a complete shape is not a new idea; it has been used in 3D human reconstruction like [a] many years ago. Although such an idea has not been proposed in the form of 3DGS, I don't think it is novel enough to be claimed as a technical contribution. \n\n* It is unclear whether the baseline methods are trained on the same dataset as the proposed method or not. In Line 244 of the main paper, the authors mention that they \"use the official implementation\", but it is still unclear whether the authors use the pretrained network weights or retrain these baselines using the same dataset.\n\n[a] Gabeur et al. Moulding Humans: Non-parametric 3D Human Shape Estimation from Single Images. ICCV 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and helpful comments, which triggered deeper thinking about our approach. We would like to address your concerns in the following sections:\n\n**How could the static points produced by the reconstruction branch model the dynamic expressions?**\n* Your understanding is mostly correct. In our method, the reconstruction branch generates static Gaussian points, while the expression branch generates dynamic Gaussian points.\n* However, it's important to note that our Gaussians are not purely RGB or spherical harmonics Gaussians. Instead, our Gaussians include 32-D features (as described in Sec. 3.3). In Fig. 7 of the paper, we visualize the first 3 dimensions of these features (i.e., the RGB values of the Gaussians) without the neural rendering module. This visualization is intended to intuitively display the functionality of each part and the importance of each branch should not be judged based on RGB values ​​alone. Their importance is determined by the entire 32-D features and the neural renderer module. In fact, the expressions are modeled solely by the expression branch, without expression Gaussians, the lifted Gaussians from the reconstruction branch are static.\n\n**How does the proposed method resolve the potential conflicts of these two point sets?**\n* Thank you for pointing out some missing parts of the paper. We will include this discussion on how to resolve conflicts in the revised version. \n* Although we attempt to bring the two point sets closer together, there are inherent conflicts since one set is static and the other is dynamic. We address these conflicts through the neural rendering module. Our Gaussian points have 32-D features, which contain more information than just RGB values. Thus, the neural rendering module can leverage these features to integrate the two point sets. We show some results with conflicts in Fig. 4 on the new supplementary page. It can be seen that when there are significant expression differences, the RGB values of the Gaussian points conflict, but these conflicts are well-resolved after neural rendering. We believe this demonstrates that the neural rendering module acts as a filter, using features from the expression Gaussian in areas where conflicts may arise.\n\n**About the novelty of the dual-lifting method.**\n* We believe our method is novel. The work of [Gabeur et al.] uses two sets of points to model a static human body and then obtains the body mesh. Our method uses two sets of lifted 3D Gaussians to model the human head, which is the first one-shot 3DGS method for head avatars. Additionally, we incorporate expression Gaussians, enabling dynamic expression modeling, which further develops this method. Our method also achieves real-time re-driving, which is a first in one-shot head avatars and is crucial for practical applications. Finally, our work provides a comprehensive evaluation and comparison for one-shot dynamic head avatar reconstruction, which builds a solid baseline for future research.\nFor missing citations and discussion of related papers, we will include and discuss them in the revised version.\n\n**The baseline methods and evaluation method.**\n* We used pre-trained network weights and the official code implementations for evaluation. Since each method has different data requirements, it is challenging to train every method on a unified dataset. For instance, Portrait4D uses single-view head images, Portrait4DV2 uses generated multi-view data, GOHA needs video and single-view head images, Real3DPortrait needs EG3D-generated multi-view head images and pre-train image-to-plane model, ROME did not release their training code. \n* Since these methods emphasize their generalization ability, using pre-trained weights for evaluation is acceptable and all compared baseline works also follow the same way for their evaluation.\n* Additionally, the HDTF and VFHQ test sets are commonly used benchmark datasets. OTAvatar, GOHA, and GPAvatar have reported results on the HDTF test set, while GPAvatar, Portrait4D, and Portrait4DV2 have reported results on the VFHQ test set, therefore we chose to test on these two datasets.\n\n**Side-by-side video comparison with baselines.**\n* Due to rebuttal restrictions, we are unable to provide additional videos for rebuttal. Instead, we have included some consecutive frames in the new supplementary page and highlighted the areas of interest. We will add side-by-side video comparisons in our future open-source code and demo website.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Firstly, we would like to thank all the reviewers for their thorough review and valuable suggestions. We summarize the issues raised and indicate where we address them. Additionally, we provide some new visual results in the supplementary page.

We answered the following questions in our response to **PdVU**:
* Robustness of the model to image quality, lighting condition, occlusion, and extreme expression. (**PdVU**)
* How we ensure the diversity of training data. (**PdVU**)
* The complexity of model implementation and the resource consumption during the training process. (**PdVU**)

We answered the following questions in our response to **3QbB**:
* Why don’t we predict offsets for expression Gaussians? (**3QbB**)
* Do the lifted Gaussians and expression Gaussians contract each other during expression? (**3QbB**)
* Why don’t we bind the lifted Gaussians to the 3DMM expression Gaussians? (**3QbB**)
* Why do we use only 3DMM vertices? (**3QbB**)

We answered the following questions in our response to **65B2**:
* Ablation study of removing the reconstruction branches (**65B2**)
* Explanation of quantitative results and over-smoothed qualitative results. (**65B2**)
* Discussion on the impact of 3DMM accuracy on model performance. (**65B2**, **PdVU**)
* There should be some failure cases to show the limitations of the work. (**65B2**)

We answered the following questions in our response to **d3t7**:
* Further explanation of the model design, including how we model dynamic expressions. (**d3t7**)
* How we resolve conflicts between the two sets of Gaussian points. (**d3t7**, **3QbB**)
* Further discussion of the novelty of our method. (**d3t7**)
* Details on how we conduct our evaluation and comparison. (**d3t7**)

We also show more results in the new supplementary page:
* Side-by-side video frames comparison results and qualitative results with highlights. (**3QbB**, **d3t7**) Due to page limitations we selected several of the most competitive baseline methods to show video frames.

Meta-review: This paper proposes a method for one-shot 3D reconstruction of faces from single input images along with their animation using input FLAME expression parameters. The authors propose a dual Gaussian lifting framework to lift a static head and the expression coefficients into a Gaussian representation. Additionally for the static branch, a backwards and forwards lifting approach is proposed, which is novel. A neural render is further proposed to resolve any discrepancies between the dynamic and static branches. The proposed approach also relies heavily on the parametric 3DMM model for regularizing 3D reconstruction. The proposed algorithm is compared to several existing state of the art baselines on two benchmark datasets and shown to be better in terms of reconstruction accuracy and expression preservation, but not SOTA in terms of identity preservation. By using Gaussian splatting the proposed method is also fast in terms of training and inference times versus the existing approaches. Lastly, without requiring test-time fine-tuning to new subjects the method generalizes out of the box to unseen faces.

The reviewers appreciated the novelty of the proposed approach in being the first to employ Gaussian splatting for generalized one-shot animatable avatar creation, its speed, generalization ability and the quality of the results. However, reviewers raised concerns about the heavy reliance on a 3DMM, which does not allow for parts of the face that aren't mapped to the 3DMM (hair and tongue) to be animatable. Requiring FLAME fitting, also makes identity and expression parameters to not be disentangled fully. Reviewers also raised concerns about the design choice of employing the dual independent dynamic and static branches both if which predict Gaussians, which could lead to differing shapes and hence is not an intuitive design. 

All things considered, the AC feels that the proposed approach is a step in the right direction towards employing 3D Gaussians for generalized animatable 3D head avatar creation.While not perfect, it would form a strong baseline for future work to build upon and encourage further exploration and improvements w.r.t. moving away on the reliance on 3DMM. The authors have also promised to release their code. Hence, the AC leans towards accepting this work.
Decision: Accept (poster)