Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis
Authors: Zhenhui Ye
Tianyun Zhong
Yi Ren
Jiaqi Yang
Weichuang Li
Jiawei Huang
Ziyue Jiang
Jinzheng He
Rongjie Huang
Jinglin Liu
Chen Zhang
Xiang Yin
Zejun MA
Zhou Zhao
Venue: ICLR 2024 spotlight (2024)
Keywords: One-shot Talking Face Generation
Neural Radiance Field
Primary area: generative models
TL;DR: We present a zero-shot NeRF-based talking face video system that could generate 3D avatar with realistic torso movement and supports both audio/video-driven applications.

Abstract: One-shot 3D talking portrait generation aims to reconstruct a 3D avatar from an unseen image, and then animate it with a reference video or audio to generate a talking portrait video. The existing methods fail to simultaneously achieve the goals of accurate 3D avatar reconstruction and stable talking face animation. Besides, while the existing works mainly focus on synthesizing the head part, it is also vital to generate natural torso and background segments to obtain a realistic talking portrait video. To address these limitations, we present Real3D-Potrait, a framework that (1) improves the one-shot 3D reconstruction power with a large image-to-plane model that distills 3D prior knowledge from a 3D face generative model; (2) facilitates accurate motion-conditioned animation with an efficient motion adapter; (3) synthesizes realistic video with natural torso movement and switchable background using a head-torso-background super-resolution model; and (4) supports one-shot audio-driven talking face generation with a generalizable audio-to-motion model. Extensive experiments show that Real3D-Portrait generalizes well to unseen identities and generates more realistic talking portrait videos compared to previous methods. Video samples are available at https://real3dportrait.github.io.

PEAL3D-PPORTTAA ONE- REALISTIC 3D
ALLKING PIRTRAL
PORTRAITSS SYNTHESIS
Zhenhai Ye Tianyun Zhong
Yi Ren & Jiaqi Yang to Weichuang Li 9 Ren Jinzheng He
Jiawei Huang
Ziyue Jiang 9
a Rongjie Huang * Jinglin Liu
Chen Zhann Xiang Yin Zejun Ma Zhou Zhao
Zhejjaang University & ByteDance & 9 HKUST(GZZ)
izhenhuiye, zhaozhou) ) Oz www..eed juujed.decc. Iren. yi, ginxiang stephen| 1 @bytedance.com coaance.cco
ABSTRACT
One-shol 3D talking portrait generation aims to reconstuud 3D avatar from an unseen image, and then animate it with reference video audio to generate a talking portrait video. The existing methods fail to simultaneously achieve the goals of accurate 3D avatar reconstruction and stable talking face animation. Be- sides, while the existing works mainly focus on synthesizing the head part, it is also vital to generate natural torso and background segments to obtain a realistic talking portrait video. To address these limitations, we present Real3ID-Potrail a frameeeee that (I) improves the one-shot 3D reconstruction power with large image-to-plane model that distills 3D prior knowledge from a 3D face generative model; (2) facilitates accurate motion-com condinnnoddi animation with an efficient motion adapter; (3) synthesizes realistic video with natural torso movement and switchable background using head-torso-bacee forso-background super-resolden model; and (4) supports one-shot audio-driven talking face generation with generalizable audio-to-mon mo-toooon model. Extensive experiments show that Real3D- Parttrat general- izes well to unseen identities and generates more realistic talking portrait vidcos compared to previous method. !!]
image-
INTRODUCTION IRTRODUCTION
Talking head generation aims to synthesize talking portrait video given the driving condition (either
a motion sequence (Wang et al., (021) or a driving audio (Chung et al., 2017 Kim et al., 2019]
Yi et al. 2020 Ye et al., 2022). }. It is long-standing cross-modal task in computer graphics and
computer vision with several real-world applications (Huang et al., 2022 2023a) such as video conferencing Huang et al. (2023b) and visual chatbot Ye et al. (202cc) Previous 2D methods (Wang) et al. 2021 Zhou et al.) 2020 Lu et al. 2021) could produce photo-realistic videos thanks to the power of generative adversarial networks (GAN). However, due to the lack of explicit 3D modeling,
confereee
these 2D methods are challenged with warping artifacts and unrealistic distortions at a significant head movement. In the past few years, neural radiance field (NeRF)-basee 3D methods (Mildenhall
ct al. 2020 Guo et al. 2 Hong et al. 2022b Shen et al., 2022, Ye et al. 2023b) have been prevailing since they maintain realistic 3D geometry and preserve rich texture details even at large head pose. However, among most of them, the model is over-fitted to specific person, which requires expensive individual training for every unseen identity. It promising to explore the task of one-shot 3D talking face generation, i.e., given an unseen person's 's reference image, we aim to lift it into a 3D avalat and animate with the input condition to obtain realistic 3D talking person video.
Tineeeee With recent advances in 3D generative models, it is possible to learn hidden space of 3D tri- plane representation (EG3D, Chan et al. (2022)) that generalizes to various identities. While recent works (i et al. 2023b; 2023) have pioneered one-shot 3D talking face generation, they fail
'Equal contribution.
I Interns ByteDance.
I *Corresoondng author. VVdeo samples and source code available at hwttthttpt//eeee ttttp:://reaal3dortraaa 3dportrait wit.t...iittttttt.. io

---

to achieve accurate reconstruction and animation simultaneously. To be specific, some works (such as OTAvatar, Ma et al. (2023)) first generate a 2D talking face video, then lift it into 3D via 3D GAN inversion (Yin et al. 2023), which enjoy rich 3D prior knowledge from the 3D GAN, yet are challenged with animation artifacts like temporal jitters and image distortion. Another line of works (Yu et al.] 2023 Li al. 2023a) learn feed-forward network predict the 3D representation from the image, then deform the 3D model with the input condition. However, the image-to-3D reconstruction process is not robust due to the lack of large-pose multi-view frames in video datasets, which are essential for learning 3D geometry. Due to the aforementioned challenges, the existing one-shot 3D methods cannot produce high-quality talking face videos.
Based on this observation. the first goal of this paper is to improve the 3D reconstruction and animatiion power: (1) As for reconstruction, we propose to first pre-train large image-to-Plane (I2P) model by distilling 3D prior knowledge from well-trained 3D face generative model. The 12P model feed-forward network that learns to reconstruct 3D representations of the input image with a single forward. We combine the advantage of ViT (Dosovitskiy et al. 2020) and VGGNet (Simonyan & Zisserman 2014) to construct the network architecture and scale up the model to better store the knowledge of the image-to-3D mapping. (2) As for animation, Adapter (MA) to morph the predicted 3D representation given the input condition. Specifically, the motion adapter takes fine-grained motion representation, projected normalized coordinate code (PNCC) (Zhu et al.) 2016) as the input, then predicts a residual motion diff-plane that edits the reconstructed 3D representation via element-wise addition. Since PNCC well disentangled from appearance and pose, we use a shallow SegFormer (Xie et al.) (2021) I) to inject the input expression information into the canonical space efficiently. In word, we pre-train a large-scale image-to- saagge-tt-planee plane backbone for generalized and high-quality 3D reconstruction, then utilize a lightweight motion adapter to achieve efficient face animation.
design an effective facial Motion
The second goal is to improve the naturalness of the synthesized torso and background segments. Existing methods either only model the head part (Hong et al. 2022b) or model the head and torso as a whole (I.i et al.) 2023b), which overlook the necessity of a natural torso and background to obtain a realistic talking portrait video. To handle this limitation, we propose to individually model the head, torso, and background segments and compose them into the final image during the rendering process. Specifically, we design a Head- -orad-Toro--aaaa Brrko-dn Super- Repol-Resooution (HTB-SR) model, which consists of super-resolution branch to upsample the oow-resolution images, a warping- based torso branch for modeling the individual torso movement, as well as a background branch to achieve switchable background rendering. With these designs, we could render realistic and high-fidelity 3D talking portrait video given the motion condition. To further support audio-driven applications. we design a generic variational audio-to citiiottt-otioon (A2M) model to transform the audio signal into the motion representation PNCC. Our audio-to-motion model generalizes well to unseen identities without adaptation and supports explicit eye blink and mouth amplitude control.
celume-rendered head
To summarize, in this paper, we propose Real3D-Portrail one-shot and Realistic 3D talking Portrait generation method that: (1) improves the 3D reconstruction and animation power with I2P model and motion adapter; (2) achieves natural torso movement and switchable background rendering with HTB-SR model; and (3) proposes a generic A2M model, hence becomes the first one-shot 3D face system that both supports audio and video-driven scenarios. Experiments show that our method outperforms existing one-shot talking face systems and achieves comparable performance to state-of the-art person-specine methods. Ablation studies prove the effectiveness of each component.
2 RELATED WORK
Our work focuses on the task of one-shot 3D talking face generation, it mainly relates to two aspects of reconstruction and animation, i.e., (1) How to reconstruct accurate 3D face representation of the input image; (2) How to morph the 3D representation and render the talking face that corresponds to the driving condition (motion or audio). We discuss them respectively in the following sections.
3D Face Representation Introducing 3D face representation into the talking face generation is a fundamental technique to improve the naturalness of the synthesized video. The earliest adopted 3D representation is the 3D Morphable Model (3DMM) (Blanz & Vetter 1999). which provides a strong geometry prior the face rendering process. However, the accuracy of 3DMM is known
2

---

Source image
Canorical of plane
Face Parsing
Image to Plane Model
Wp
Head segment
Volume Renderer
30MM Estimator
Siuurcn barkgrmund
la
Driving resources: video audio
or
Motion Adapter
Medion difigtane
Head-Torso-Bee Barko-Bac Super Resolution model
Audio-to-
Motion
Facial motion /Source/ Driving PNCC)
Precicted frames
Figure I: The inference pipeline of Real3D-Portrai With one source image as input and video/audio as driving condition, synthesizes 3D talking avatars with a realistic torso and background.
to be unsatisfactory due to two limitations: (1) the reconstructed face mesh is low fidelity and lacks details such as wrinkles; (2) only the face region is modeled by the parametric model, and fails to represent other regions, such as hair, hat, and eyeglass. Then, a 3D head representation based on Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020) emerged. Early NeRF-based 3D -e-person--er-traiinng manner, which takes
representations are typically extracted an inefficient tens of hours to fit each identity. Recently, the invention of tri-plane representation (Chan et al. 2022)
and its usage in 3D face GAN paves the way for high-quality and efficient NeRF-based 3D face reconstruction. Some works (Sun et al. 2023) utilize GAN inversion (Roich et al.) (0022) to obtain tri-plane from the pretrained 3D Face GAN, which suffers from slow inference and temporal jittering. image-to-plane
In contrast, other predictor-based works (Trevithick al., 2023) explore learning
mapping that directly maps the input image to the tri-plane representation, which is more efficient and stable during inference. Our large image-to-plane model follows the predictor-basee paradigm.
The earliest 2D-based face animation methods like (Prajwal al. 2020 Hong et al. 2022a) directly adopt GAN to generate the result, which results in training instability and bad visual quality. Later, the warping-baae based methods (Siarohin et al. 2019 Wang et al. 2021 Zhao
2D/3D Face Animation
& Zhang 2022 Pang et al. 2023 Hong & Xu 2023) aim to warp the pixels of the source image with a dense warping field given the D--aware key points extracted from the driving video. achieves
high image fidelity, yet due to the absence of a strict 3D constraint, it is challenged when driven
large head pose and suffers from warping artifacts and distortions. To handle the artifacts caused by 21) modeling,
works resort 3D-based methods.
The earliest 3D talking face methods are primarily based on the 3DMM, which typically first performs 3D reconstruction of the input image (Deng al. 2019c Daněček al. (0022), then incorporates the 3DMM prior into the face rendering process (Wu et al. 2021). However, these methods fail to generate photo-realistic results due to the information loss caused by 3DMM. Recently, NeRF-based talking face generation has prevailed since it combines the advantages of high image fidelity and (Guo et al.) 2021 Tang al.) 2022 Ye et al.) 2023a), which requires tens of hours of individual training for every unseen identity. Most recently, some works explore one-shot NeRF-based talking face generation with the tri-plane representation. which can be categorized into two classes. The first class (Ma et al., 2023 Li 2023 Trevithick et al.) 2023) adopt a 2D animation and 3D lifting pipeline, which utilizes a bre-trained 2D talking face system to obtain 2D talking face video, then lift it into 3D via iterative 3D GAN inversion (Yin et al. 2023). This line work could enjoy the robust 3D prior knowledge of a pre-trained 3D GAN is also challenged by the unstable GAN inversion and degraded performance at large head poses. The second class adopts a 3D reconstruction and 3D animation pipeline (Li al. 2023a b). which learns an image encoder to predict the 3D representation, then morphs the reconstructed 3D model given the condition. However, since video datasets typically lack large-view frames, the generalizability of 3D reconstruction is unsatisfactory. Due space limitation, we discuss the relationship between our approach and previous methods in Appendix A
strict geometry constraints. However, most of the successful ones are identity-overfine
REAL.3D-PORTRI
Real3D-Portrain aims to achieve realistic one-shot video/audio- driven 3D talking face generation. As shown in Fig. the overall inference pipeline is composed of a large image-to-plane (I2P) model
3

---

Split
Tri-plane
plane
230x2Fe006
Fulliaage 512H51EK3
Conv20
Split
Output map
Leak8alU
X
Upsample
XGrz
Linear
Source Background
SR Branch
Upsample2X
Concat
LayrerNorm
512-512k3
253
Linear
Interpolate
Concat
Background Branch
Conv20 LeakReLU
X
Interpolate Concet
Segt-rrmer Block f
LayerNorm Attention
Conw2D DownSample 2X
Segformer
Overlap PatchEmbed
Tarsa
Block
Segf Former
VIT Branch
Block
Source / target keypolnnss
Input feaature map HoWWKG
Source/ Target
Input image
PNCC
Predicted 128kx 12fcc
55zw5tba3
(a) Image-to-Plan Plaane model
(b) Motion Adapler
(c) SegForeer Block
(d) HTB-SR model
Figure 2: The network structure of I2P model, motion adapter, and HTB-SR model.
(Sec. 3.1 to reconstruct a 3D head representation and motion adapter (Sec. 3.2 to morph the 3D head given the facial motion. Then, we could render the head image at an arbitrary camera (head) pose with the volume renderer. Afterward, (HTB-SR) model (Sec. 3.3) to synthesize the final image at 512x512 resolution with individually modeled torso and background. To support audio-driven applications,
super-resolution propose head-torso- hecadttors--backrroum
also design generic audio-to-motion (A2M) model (Sec 3.4) to transform the raw audio into the corresponding facial motion. The training process of the four models is sequential. We describe the designs and training process in detail.
3.1 IMAGE-TO TO-NE MODEL FOR 3D FACE RECONSTRUCTION
In the first stage, we need to reconstruct a canonical 3D face representation Peano of the target identity in the source image Tue. Specifically, learn feed-forward network that directly transforms the input image into the tri-plane representation, namely the Image-to-Plane (I2P) model.
Network Design When designing the network structure of the I2P model, we notice two main challenges for the network: (1) should map the input image to canonical tri-plane, which requires coordinate transform from pixel coordinate to world coordinate. (2) It should extract rich appearance features from the source image to guarantee the fidelity of the rendered image. To this end, as shown in Figure 2 a), we design hybrid network consisting of a VIT and VGG branches. The ViT branch comprises a stack of SegFormer blocks, which exccutes attention among patches and could efficiently handle the pixel-to-word due to the patch embedding operation, as a complementary, we design VGG branch, which is simply stack of convolution layers extract high-frequency appearance features. The branches' output information is fused via concatenation and further processed with shallow convolution layers to produce the final tri- tri-plane representation. Note that we remove all normalization in the VGG branch to keep the
canonicalization process. Since VIT cannot maintain high-frequency texture
appearance-related bias in all hidden layers.
Pre-training Process The talking face dataset typically lacks multi view frames, which nec- essary for the model to learn 3D prior knowledge. To improve the generalizability of 3D face reconstruction, inspired by the multi-view reconstruction task proposed by Trevithick al.) (2023). we first pre-train the I2P model on multi-view image dataset synthesized by EG3D (Chan et al. 2022), 3D GAN for generating human face. We illustrate the pre-training process in Appendix B.
3.2 MOTION ADAPTER FOR 3D FACE ANIMATION
With the pre-training process of the I2P model, achieve to reconstruct accurate 3D face representation from the source image. Then, we train a motion adapter to animate the predicted 3D face, given the input motion condition.
4

---

Reference image 51ZK512k3
Refereice head image 512x512x3
Finetuned from
Canonical tripplane 256x25hxc224
Adv loss
Face
Image-to Plane
Target image $12x5122x3 2xc3
Volume
Target PNCC
video
3DMM mn Estimator
Motion Adapter
LI Joss
TX5L2x512>3
Motion diff plane
VGGFace loss
loss
Target
256o256x612c1
Face Parsing
Tanget head 61260122
Fixed Part
Learnable Part
Figure 3: The process of training the motion adapter and fine-tuning the I2P model Sec 3.2
Motion Representation We use the projected normalized coordinate code (PNCC) Zhu et al. 2016; Kim al. 2018 Li et al. 2023a) the motion representation, which is posc/appearance- agnostic feature map that possesses fine-grained facial expression information based on a 3DMM face. Specifically, given a pair of identity code i and expression code c, we could obtain the PNCC by rasterizing the 3DMM face mesh at the canonical pose through Z Z-Buffer (Phong) 1998) algorithm with NCC (Zhu et al. 2016) as its colormap. We provide details of calculating PNCC Appendix
B.2
Thanks the
decomposition of 3DMM, we can utilize PNCC to achieve identity- agnostic motion-conditioned face animation. During training, we first fit the 3DMM parameters of the training video to obtain the ground truth PNCC. During inference, we could construct the driving PNCC by:
PNCC = Z-Buffer(3DMMM LeeaMMM_eshiig--e Car NCCC)
(1)
where iae the identity coefficient of the source image, and Carv is the expression coefficient that is either extracted from a driving video or predicted by an
model (Sec. 3.4).
Predicting Residual Motion Diff-plane Once the motion representation is decided, the second question is how to inject the motion condition into the 3D representation to control the facial expression. We do not choose the deformation field as previous works do since it typically results in bad quality of the predicted mesh (Li et al. 2023a). Instead, given well-trained I2P model to produce 3D representation that possesses accurate geometry/texture information, we propose to learn light- Motion Adapter (MA) predict residual motion diff-plane air that only edits the minimal geometry change of the canonical tri-plane Pcano based on the different motion condition. As for the network structure of the proposed MA, as shown in Fig. bb),
adopt shallow
SegFormer (Xie et al.] [021] to enjoy its high efficiency and strong ability to achieve cross-coordinate transform brought by the attention over feature map patches. To be specific, the process of animating the source image I given the input motion condition PNCCar and camera pose cam can be expressed as:
Iarv SR(VR(P camo + Pair cam)), s.l., P Peano I2P Pai MA(PNCC PNCC
(2)
where VR and SR are vanilla volume renderer and super-resolution module, whose structures are shown Fig. 5) I2P and MA are the proposed image-to-plane model and motion adapter, respectively; PNCCar are the driving motion representation defined in Eq. and PNCC e the PNCC extracted from the source image. Note that the input of MA is the concatenation of PNCC and PNCCar. It since the tri-plane predicted by the I2P model is of the source expression and the MA should be aware of both the source/driving expression to correctly map the 3D representation of the source expression into the target expression.
Training Process
We illustrate the training process at this stage in Fig. 3 Apart from learning the motion adapter from scratch, we also fine-tuned the I2P model and the VR/SR module from the large-scale and high-fidelity talking face video dataset (Zhu et al. 2022). To construct the training data pair, we randomly select two frames from video and define them as source image I and target image L Since camera pose is only highly correlated with the movement of the head part, we only consider the head region at this stage, so all images preprocessed with face parsing model extract the head segment. The source
pre-trained weights in Sec. 3.1 We train the model
5

---

head image is fed into the I2P model to reconstruct canonical tri-plane P cano. and the PNCC extracted from Lgg is fed into the motion adapter MA to obtain the residual motion diff-plane Pait-
Then we could obtain the predicted image I' igi following Eq. 2 The training loss is as follows:
L tgr I' l'gllll i + Lvacs (Tys, I' u') + DumllAdy (T(y)) + Lrap
where the first two terms are L1 loss and VGG19/VGGFace-
(Simonyan & Zisserman 2014
Parkhi et al. 2013) perceptual loss; LimiAAd is the dual adversarial loss proposed by (Chan et al.
2022) to improve the image fidelity and consistency; and Li is our proposed Laplacian loss over
the motion diff-planee of adjacent frames, which acts as a regularization term to eliminate temporal
jittering. To be intuitive, given PNCCs from frame {1 10-1,,1 1,4,4 + 1}, we expect the diff-plane Pan of frame to be the average of that of - 1 and + 1: Lap |M((PNCC)) 0.5 (MA(PNCCC-)) MA(PNCC
(4)
3.3 HEAD-TORSO- HAAD-TORSOOBBAKGRRUUN BACKGROUND SUPER-RESOL RESOLUTION MODEL
With the proposed I2P model and motion adapter, we could synthesize 3D talking heads given the source image and driving motion. The last step towards a realistic talking person video is synthesizing the torso and background segments. A naive solution is to model the torso/background along with the head using NeRF, However, applying the same rigid transformation to both the head and other regions results in unsatisfactory results (e.g., the torso and background will rotate along with the head movement). To generate realistic torso and background, we propose a Torso- Background Super-Resolution (HTB-SB SR) model to individually model the head, torso, and background segments and fuse them into high-resolution composite image.
Network Struture As shown in Fig. of Appendix B.3 the HTB-SR model consists of an SR branch, a Torso branch, and a Background branch. (1) The SR branch shares a similar structure with the vanilla SR module used in See. B.2 so we could initialize its weights from the pre-trained model. (2) As for the Torso branch, since the movement of the torso is of small amplitude and often translational, we propose to model the torso part with 2D warping-based renderer that similar to (Wang et al. 202 I. which is computationally efficient and proven robust in various scenes (Siarohin jet al., 2019 . As shown in Fig. b(b), the torso segments are warped by dense flows conditioned on predefined key points to predict the torso feature map of the target image. Note that instead of learning implicit key points in an unsupervised manner as (Wang et al., 2021), we select several key points in the reconstructed 3DMM face vertex the driving condition of the torso branch, which improves the temporal stability of the predicted torso. We provide details of the torso branch in Appendix B.3 (3) As for the Background branch, the biggest challenge
fill the pixels that were occupied by the foreground (i.e.. the person) in the source image. To this end, as shown in Fig. ().) we first adopt a nneirett-neighbor (KNN)- based inpainting method to preprocess the background segment of the source image. Then, we use shallow convolution layers to extract texture features from the inpainted background. More details about the background branch can be found in Appendix [B.4] (4) As for fusing the three feature maps of head, torso, and background segments into a composite image, we found a direct channel -ise concatenation leads to hollow artifacts and blurry results in the boundary region (shown in Fig. [6). We suppose that the problem is caused by the unlimited information propagation among these three segments' feature maps and handle this problem with a alpha-blending-styl fusion mechanism. To be specific, we first obtain the head mask Mbead and torso mask Mono- then integrate the three segments with the awareness of occlusion:
(5) where F denotes the extracted feature map, Mperson is the person mask obtained by bitwise-or operation to Maead and Morso- Details of obtaining Maead and M can be found in Appendix B..)
F= (Fhead Mhead + Fronso (1 (1-M -h-aa))) Mparson Fbg
M Mpesoo)
Training Process As shown in Fig, 7
we load the pre-trained I2P model, motion adapter, and volume renderer from Sec. 3.2 and replace the SR module with our HTB-SR model. Only the HTB-SR model is updated at this stage, and all other parameters are frozen. The training objective similar to Eq. 3 The difference is that the GT and predicted images are full images with head/torso/backgronn parts instead of only the head segment.
We can also use saute-of-hhear neural network-hased -hased inpainting methods to obtain more realistic back- ground image. But since that is not the focus of this paper, we simply used the naive KNN- method.

---

3.4 GENERIC SUDIO-TO-MOTICN MODEL
To support audio-driven applications, we design a generic and controllable audio-to-motion (A2M) model to transform the audio into the PNCC motion representation. Inspired by Ye et al. (2023b), we adopt flow-enhanced variational auto-encoder (VAE) to learn accurate and expressive audio- to-motion mapping. HuBERT (Hsu et al.] 2021) is chosen as the audio representation. As for the predicted motion representation, instead of directly predicting the PNCC, we choose predict the 3DMM expression parameter, which utilizes the strong geometry prior of 3DMM and significantly improves the training stability and audio-lip accuracy. We choose BFM 2009 (Paysan et al. 2009) as the 3DMM model. Since all expression bases are orthogonal, given the same identity code, the reconstructed 3D face meshes in a video are uniquely determined by the expression code. Hence an L2 error on the expression code, LExpRecon is feasible to be the reconstruction term in training the VAE. To encourage the model better reconstruct the facial landmark (instead of only the 3DMM parameters), we additionally introduce the I.2 reconstruction error of 468 key points of the reconstructed 3DMM vertex, LddmReoon as an auxiliary supervision signal. The training loss of the generic audio-to-motion model is as follows:
LA2M = CxL + LExpRecon + LddmReoon + ExpLaap
(6)
where CKL is the KL divergence of VAE; LexpLap is the laplacian loss of the predicted expression code sequence to climinate temporal jittering. To further improve the controllability, we add the eye blink and mouth amplitude the auxiliary condition to the A2M model, which improves the expressiveness of the generated video. We provide detailed structure of A2M model in Appendix
Seltio/Videe-Drieen Inference Once the four-stage training process is done, no further training is required for a new identity. During inference, as shown in Fig. [ we first fit the 3DMM parameters of the source image to obtain the source identity code i As for audio-driven scenarios, the expression sequence Bärv corresponding to the input audio using the A2M model; as for the video-driven scenarios, fit 3DMM on the reference video obtain the earv- Then, we could obtain the driving PNCC following Eq. | and render the final images following Eq. annd Eq. S
obtain
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
Implementation Details. We provide detailed configuration and hyper-parameters in Appendix C and will release the source code at
github. in the future.
Data Preparation. To pre-train the I2P model, we adopt 3D face generative model (Chan et al. 2022) to on-line generate multi-view image pairs during training. To train the motion adapter and HTB-SR model, we use a high-fidelity talking face video dataset, Celeb (Zhu et al. (022). which is about 65 hours and contains 35,666 video clips with a resolution of 512x512 involving 15,653 identities. To train the A2M model, use VoxCeleb2 (Chung et al. 2018), low-fidelity but 2.000-hour-long large-scale lip-reading dataset to guarantee the generalizabiling of the audio-to- motion mapping. We preprocess the video frames with an off-the-shelf landmark extractor and face parser (Lugaresi et al. 2011)) then fit 3DMM parameters based on the projected landmark error. We extract HuBERT features (Hsu et al. 202T) and pitch contours from the audio track.
Compared Baselines. We compare our ReataD-PPtrrit with several video/audio-driven baselines: 1)
Fice-vid2vid (Wang et al. 2202)), a widely used warping- based video-driven talking face system; 2)
OT-Avatar (Ma et al.] 2003), recent one-shot video-driven method that utilizes pre-trained 3D
GAN to obtain 3D talking video; 3) HiDe-NeRF (Li al. (202a)), a state-of-the-art one-shot 3D)
talking face system that utilizes deformation field for face animation; 4) MakeltTalk (Zhou et al.) 2020 and 5) PC-AVS (Zhou et al. 2021), which are two one-shot audio-driven talking face method that achieve good audio-lip synchronization; 6) RAD-NeRF (Tang et al. 2022), a NeRF-based method that achieves high-realistic quality by over wettiiiting on the target person video. Note that it is unfair to compare RAD-NeRF against other one-shot methods, but we compare with it to show how far we are from the performance of the state-of- thethheart person-speciff method. We summarize the characteristics of all test baselines and our method in Table. 5
7

---

Table I: Same/Cross-identity reenactment results of video-driven methods. Best scores are in bold.
Same-Identity Reenactment
Cross-Identity Reenactment
Methods
L1.
PSNR 1
SSIM 1 I.PIPS 4 FID L CSIM
AED L
APD J AKD
CSIM f FID /. AED 4 APD 4
Face-vid2vid
0.078
16.47
0.779
0.184
42.%
0.808
0.116
0.023
3.176
0.726
45.18
0.144
0,029
OTAvvatar
0.106
13.17
0.672
0.232
65.37
0.568
0.170
0.040
5.891
0.544
64.28
0.195
0.046
HiDe-NeRF
0.084
15.92
0.752
0.189
50.04
0.753
0.129
0.021
3.531
0.699
53.28
0.161
0.025
Ours (VD)
0.067
18.95
0.801
0.171
37.50
0.821
0.111
0.018
2.829
0.758
42.37
0.138
0.022
4.2
QUANTITATIVE EVALUATION
Real3D-Portrae supports both video and audio as driving resources. In this section, we evaluate and the baselines for video-driven reenactment and oudio-driven talking face generation, respectively.
Video-driven same/cross-ident identity reenactment. In the video-driven (VD) scenario, the driving motion condition and head pose are obtained from reference video. Under the same-identity setting, we use the first frame of the reference video as the source image; otherwise, the source image is of a different identity. As for the same-identity setting, we evaluate PSNR, SSIM, cosine similarity of the identity embedding (CSIM) by Deng et al. (2019)), average expression distance (AED) and average pose distance (APD) based on (Deng et al. 2019b), average keypoint distance (AKD) based on (Bulat & Tzimiropoulos 2017), well as LPIPS. LI and FID between the reenacted and ground truth frames. As for the cross-identity setting, since there is no ground truth, we evaluate the results based the CSIM, AED, APD, and FID metrics. The results are shown in Table T Firstly, our video-driven Real3D-Portren performs best in terms of LI, PSNR SSIM, LPIPS, and FID, hence achieving the best image quality. Secondly, our Real3D-Portrait gets the highest CSIM, which denotes that it preserves the identity of the source image. Finally, our method achieves the best AED, APD, and AKD, demonstrating that ours could accurately animate the 3D avatar given the input condition.
Table Results of audio-driven methods
Audio-driven talking face generation. In the audio-driven (AD) setting, the driving mo-
Methods
CSIM
FID|
AED
Synct
tion conditions predicted from the input audio. Similar to the cross-identity reenact-
MakellTalk
0.715
32.65
0.213
3.286
PC-AVS
0.327
82.02
0.162
6.483
ment, since there are no ground truth samples, we use CSIM to measure the identity preser- vation, FID to measure the image quality, and
RAD-NeRF
0.784
39.45
0.197
3.779
Ours (AD)
0.763
43.02
0.146
6.565
AED and Syne score (Prajwal et al. (020) to measure the audio-lip synchronization. The results are shown in Table 2 When compared with MakeltTalk and PC-AVS, two one-shot 2D methods, our method shows significantly better identity similarity (CSIM), image quality (FID), and lip- synchronization (AED and Sync score). When compared with RAD-NeRF, a person-specific 3D method that over-fits an individual model on the tested identity's 's 3-minute-long video, apart from achieving better lip synchronization, our method remarkably shows comparable image quality and identity preserving thanks to the well-trained large I2P model and motion adapter. To summarize, the experiment demonstrates that our one-shot Peattl- outperforms other one-shot haselines and could perform closely to the SOTA person-specific over-fitting method.
4.3 QUALITATIVE EVALUATION
Case Study To make a clear comparison among all tested methods, we provide demo videos at bbtpps::///
1o We also provide more visualization results in Appendix D.3 Specifically, (1) we showcase the overall qualitative comparison of our Real3D- Portrait and other VD/AD baselines in Fig. 9 and I0 We also provide examples to show that: (2) how PNCC animates the 3D avatar in Fig. I. (3) we achieve natural torso movement under
large head poses in Fig. 12] (4) we support switchable background in Fig. [3] (5) our generic
audio-to-motion model predicts synchronized lip motion in Fig. 14
User study We conduct user studies to test the quality of generated samples. Specifically, we sample 10 audio clips from different languages and ten different identities for all methods to generate the videos and then involve 20 attendees for user studies. We adopt the Mean Opinion score (MOS) rating protocol for evaluation, which is scaled from 1 to 5. Following (Chen et al. 2020) The attendees required to rate the videos from three aspects: (1) identity preserving; (2) visual quality (including
image fidelity and temporal smoothness); (3) lip synchronization. Detailed user study settings are in
Appendix D.2. We compute the average score for each method, and the results are shown in Table
8

---

Table 3: MOS score of different methods. The error bars are 95% confidence interval.
Methods
Face-v2v
OTAvatar
HiDe-NeRF
ours (VD)
MakeitTalk
PC-AVS
RAD-NeRF ours (AD)
ID. Preserving
3.79++0.34
3.29++024
3.74+0.31
4.08+00.1
3.38+0,44
3.21+0036
4.12=0.26
4.05++0.27
Visual Quality
3.73+0.25
3.28+0.29
3.45+0.22 4.16+00.3
3.34+0.34
3,40+-037
4,25+0,24
4.14+0.29
Lip Syne.
3.97=0.20 3,80+0.28
3.5440.32
4.13:10.29
2.96:0.37
4.04=0.31
3.18=0.52 4.08
3
We have the following observations: 1) Real3ID-Portren has better identity- preserving power and visual quality than previous one-shot methods and performs closely to person-specific methods (RAD-NeRE). 2) As for the lip-synchronization, Real3D-Portrait shows obvious superiority over the person-spectie audio-driven method RAD-NeRF thanks to the powerful generic mudio-to-motie model. Besides, among the video-driver methods that use GT motion as input, ours achieves the best lip synchronization, showing the effectiveness of our motion adapter to animate the avatar given the input motion accurately.
4.4 ABLATION STUDIES
Table 4: Ablation studies.
I2P and motion adapter We test four settings on the I2P and motion adapter: (1) w/o pre- training, which does not pre-train the 12P model
Methods
CSIM
FID
AED
APD1
on the multi-view image dataset in Sec 3.1 (2) w/o fine-tuning, which fix the pre-trained 12P model when training on the video dataset in Sec (3) small/large I2P model size, which tries I2P backbone of difference scales of 40M and 200M parameters (note that the default set- ting is 87M parameters); (4) w/o Lap loss, which
w/o pre-train
0.487
65.32
0.181
0.031
w/o finetune
0.683
49.21
0.233
0.027
w/ 40M params
0.725
45.48
0.140
0.026
w/ 200M params
0.754
43.15
0.143
0.023
w/o Lap loss
0.748
42.66
0.158
0.024
w/ unsup. KP.
0.746
44.86
0.138
0.023
w/ concat
0.737
46.38
0.144
0.025
w/o inpaint
0.744
43.95
0.140
0.022
removes the laplacian loss in Eq 3.2 We show the result in Table As shown in line 1 and Fig.
Full (VD)
0.758
42.37
0.138
0.022
15 of Appendix. [D.]] without the pre-training process, the identity similarity, image quality, and expression accuracy drop significantly. Besides, as shown in line 2, fine-tuning is also necessary to achieve a low AED. We suspect it is because the pre-trained I2P only learns to reconstruct the 3D avatar with the source expression, and it needs further updates to support face animation given the target expression. Besides, there is domain gap between the image dataset and the video dataset, hence fine-tuning is necessary to achieve better visual quality. As for the I2P model scale, as shown in line and 4, we found that 87M achieves significantly better image quality than 50M, while the performance difference between default settings to 200M is not obvious. In line 5, we found laplacian loss necessary to improve motion-conditioon head animation.
HTB-SR We test three settings on the HTB-SR model: (I): w/ unsup. KP., which is similar to Face-vid2vid that jointly learns a predictor to extract unsupervised driving key points from the predicted head image: (2) w/ concat, which replaces the proposed beaha-beendigg-- style fusion module in Eq a naive thannel-wise concatenation of the saad/torso/bcckroumm feature maps; (3) w/o impuins, which removes the KNN-based inpainting of the background image. The results are shown in Table In line 1, we can observe that unsupervised key points lead to worse visual quality due to the instability of the extra predictor network. In line 2, we find that alpha-blending-ae style fusion is necessary to obtain a good identity preserving and image quality and climinate the artifacts shown in Fig. of of Appendíx. D.4 In line 3, we find removing the background inpainting process results in worse image quality.
5 CONCLUSION
In this paper, we propose a framework for one-shot and realistic 3D talking portrait synthesis, namely Real3D-Portrait, Our method simultaneously achieves accurate 3D avatar reconstruction and animation by designing a pre-trained large Image-to-plane model and a PNCC-conditioned motion adapter. Thanks to the proposed HTB-SR model, our method is also the first one-shot 3D) method that could generate realistic video with natural torso movement and switchable background. Besides, with the introduction of a generic audio-to-motion model, our method is the first work that supports video/audio-driver applications. Extensive experiments demonstrate that our method surpasses state-of-the-an baselines from the perspective of identity preserving, visual quality, and audio-lip synchronization. Due to space limitations, we discuss limitations and future works in Appendix

---

6 ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant No. 62222211 and National Key R&D Program of China under Grant No. No.220222D01620000.
ETHICS IMPACTS
Real3D-Portrral facilitates one-shot and realistic 3D talking portrait synthesis. With the development of talking face generation techniques, it is much easier to synthesize talking human portrait videos. Under appropriate usage, this technique could facilitate real-world applications like virtual idols and customer service, improving the user experience and making human life more convenient, However, the talking face generation method can be misused in deepfake-relater usages, raising ethical concerns. We are highly motivated to handle these misusage problems. To this end, we plan to include several restrictions in the license of Real3D-Portrait, Specifically, (1) we will add visible watermarks to the video synthesized by Real3D-Portraal so that the public easily tell the fakeness of the synthesized video. (2) The synthesized videos should only be used in educational or other legal usages (like online courses), and any abuse will take responsibility by tracking the method come up with in the next point. (3) We will also inject an invisible watermark into the synthesized video to store the information of the video maker so that the video maker has account for the potential risk raised by the synthesized video.
REFERENCES
V Blanz and T Vetter. A morphable model for the synthesis of 3d faces. In 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 1999), pp. 187-194. ACM Press, 1999.
Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d 3d face alignment problem? (and dataset of 230,000 3d facial landmarks). In International Conference on Computer Vision, 2017.
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3d generative adversarial networks. In CVPR, pp. 16123-16133, June 2022.
Lele Chen, Guofeng Cui, Ziyi Kou, Haitian Zheng, and Chenliang Xu. What comprises a good talking-head video generation?: A survey and benchmark. arXiv preprint arXiv:2005.03220 2020.
Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that? arXiv preprint
arXiv:1005.02966, 2017.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleh2: Deep speaker recognition. arXiv preprint arXiv:1806.05620 05622, 2018
Radek Daněček, Michael Black, and Timo Bolkart. Emoca: Emotion driven monocular face capture and animation. In CVPR, pp. 20311 203122 2022.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 4690-4699, 201 9a.
Yu Deng, Jiaolong Yang. Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In CVPRW, 2019b.
Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruction with weakly-supervised learning: From single image image set. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 0-0, 2019c.
10

---

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Yudong Guo, Keyu Chen, Sen Liang. Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In ICCV, pp. 5784 5794,, 2021.
Fa- Ting Hong and Dan Xu. Implicit identity representation conditioned memory compensation net- work for talking head video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 3397-3406, 2022a.
Yang Hong, Bo Peng, Haiyao Xiao, Ligang Liu, and Juyong Zhang. Headnerf: A real-time nerf- based parametric head model. In CVPR, pp. 20374-20384 June 2022b.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Progressive fast diffusion model for high-quality text-to-speech. Proceedings of the 30th ACM International Conference Multimedia, pp. 2595-2605, 2022.
Rongjie Huang, Mingze Li, Dongchao Yang. Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. arXiv preprint arXiv:2304. 12995, 2023a.
Rongjie Huang, Huadai Liu, Xize Cheng, Yi Ren, Linjun Li, Zhenhui Ye, Jinzheng He, Lichao Zhang, Jinglin Liu, Xiang Yin, al. Av-transpeech: Audio-visual robust speech-to-speech translation. arXiv preprini arXiv:2305. 15003, 2023b.
Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt. Deep video portraits. ACM transactions on graphics (TOG), 37(4):1-14, 2018.
Hycongwoo Kim, Mohamed Elgharib, Michael Zollhöfer, Hans-Peter Seidel, Thabo Beeler, Christian Richardt, and Christian Theobalt. Neural style-preservve visual dubbing. ACM Transactions on Graphics (TOG), 38(6):1-13, 2019.
Shaoxu Li. Ophavatars: One-shot photo-realistic head avatars. arXiv preprint arXiv:3307.09153. 2023.
Weichuang Li, Longhao Zhang, Dong Wang, Bin Zhao, Zhigang Wang, Mulin Chen, Bang Zhang, Zhongjian Wang, Liefeng Bo, and Xuelong Li. One-shot high-fidelity talking-head synthesis with deformable neural radiance field. In CVPR, pp. 17969-17978, 2023a.
Xueting Li, Shalini De Mello, Sifei Liu, Koki Nagano, Umar lqbal, and Jan Kautz. Generalizable one-shot neural head avatar. arXiv preprint arXiv:2306.08766 2023b.
Yuanxun Lu, Jinxiang Chai, and Xun Cao. Live speech portraits: real real-time photorealistic talking- head animation. ACM Transactions on Graphics, 40(6):1-17, 2021.
Camillo Lugaresi, Jiuqiang Tang. Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv: /906.08 /8, 2019.
Zhiyuan Ma, Xiangyu Zhu, Guo-Jun Qi, Zhen Lei, and Lei Zhang. Otavatar: One-shot talking face avatar with controllable tri-plane rendering. In CVPR, pp. 16901-16910, 2023.
11

---

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, pp. 405-421. Springer, 2020.
Youxin Pang, Yong Zhang, Weize Quan, Yanbo Fan, Xiaodong Cun, Ying Shan, and Dong-ming Yan. Dpe: Disentanglement of pose and expression for general video portrait editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 427-436, 2023.
Omkar Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In BMVC 2015- Proceedings of the British Machine Vision Conference 2015. British Machine Vision Association, 2015.
Paysan, R. Knothe, B. Amberg, S. Romdhani, and T. Vetter. A 3d face model for pose and illumination invariant face recognition. Proceedings of the 6th IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS) for Security, Safety and Monitoring in Smart Environments, 2009.
Bui Tuong Phong. Illumination for computer generated pictures. In Seminal graphics: pioneering efforts that shaped the field, pp. 95-101. 1998.
KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation the wild. In ACM MM, pp. 4844492, 2020.
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Transactions on graphics (TOG), 42(1):1-13, 2022.
Shuai Shen, Wanhua Li, Zheng Zhu, Yueqi Duan, Jie Zhou, and Jiwen Lu. Learning dynamic facial radiance fields for few-shot talking head synthesis. In ECCV, pp.
Springer, 2022.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. NIPS, 32, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 2014.
Jingxiang Sun, Xuan Wang, Lizhen Wang. Xiaoyu Li, Yong Zhang, Hongwen Zhang. and Yebin Liu. Next3d: Generative neural texture rasterization for 3d-aware head avatars. CVPR, 2023.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution- robust large mask inpainting with fourier convolutions. arXiv preprint arXiv:2109.077 10..00099..11 2021.
Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition. arXiv preprint arXiv:2211 12368, 2022.
Alex Trevithick, Matthew Chan, Michael Stengel, Eric Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, and Koki Nagano. Real-time radiance fields for single-image portrait view synthesis. ACM Transactions on Graphics (TOG), 42(4):1-15, 2023.
Ting- Ming-Chum Wang. Arun Mallya, and Ming- Yu Liu. One-shot free-view neural talking- head synthesis for video conferencing. In CVPR, pp. 10039-10049, 2021.
Yaohui Wang. Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to animate images via latent space navigation. arXiv preprint arXiv:2203.091 09044, 2022.
Haozhe Wu, Jia Jia, Haoyu Wang, Yishun Dou, Chao Duan, and Qingshan Deng. Imitating arbitrary talking style for realistic audio-driven talking face synthesis. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 1478-1486, 2021.
Enze Xie. Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. NIPS, 34:12077 2021.
12

---

Zhenhui Ye, Jinzheng He, Ziyue Jiang, Rongjie Huang, Jiawei Huang, Jinglin Liu, Yi Ren, Xiang Yin. Zejun Ma, and Zhou Zhao. Geneface++: Generalized and stable real-time audio-driven 3d talking face generation. arXiv preprint xxXv:2305.00078,, 2023a.
Zhenhui Ye. Ziyue Jiang, Yi Ren, Jinglin Liu, Jinzheng He, and Zhou Zhao. Geneface: Generalized and high-fidelity audio-driven 3d talking face synthesis. In ICLR, 2023b.
Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Chen Zhang, Xiang Yin, Zejun Ma, and Zhou Zhao. Ada-tta: Towards adaptive high-quality text-to-talking avatar synthesis. arXiv preprint arXiv:2306.088 03504, 2023c.
Zipeng Ye, Mengfei Xia, Ran Yi, Juyong Zhang, Yu-Kun Lai, Xuwei Huang, Guoxin Zhang, and Yong-jin Liu. Audio-driven talking face video generation with dynamic convolution kernels. IEEE Transactions Multimedia, 2022.
Ran Yi, Zipeng Ye, Juyong Zhang, Hujun Bao, and Yong-Jin Liu. Audio-driven talking face video generation with learning-basee personalized head pose. arXiv preprint arXiv:2002.10133 20137 2020.
Fei Yin, Yong Zhang, Xuan Wang, Tengfei Wang, Xiaoyu Li, Yuan Gong, Yanbo Fan, Xiaodong Cun, Ying Shan, Cengiz Oztireli, et al. 3d gan inversion with facial symmetry prior. In CVPR, pp. 342-351, 2023.
Wangbo Yu, Yanbo Fan, Yong Zhang, Xuan Wang, Fei Yin, Yunpeng Bai, Yan-Pei Cao, Ying Shan, Yang Wu, Zhongqian Sun, et al. Nofa: Nerf-hased one-shot facial avatar reconstruction. In SIGGRAPH, pp. 1-12, 2023.
Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3657-3666. 2022.
Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose- controllable talking face generation by implicitly modularized audio-visual representation. In 2021.
CVPR, pp.
Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation. ACM Transactions on Graphics (TOG), 39(6): 1-15, 2020.
Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. CelebV-HQ: A large-scale video facial attributes dataset. In ECCV, 2022.
Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Z Li. Face alignment across large poses: 3d solution. In CVPR, pp. 146-155, 2016.
13

---

Table 5: The illustration of properties of different talking face generation methods.
Method
One-shot? 3D-Aware?
Natural Torso?
Switchable BG?
Driving Resource
Face-vid2vid Wang et al
2021
X
video
HiDe-NeRF Li ct 2023a)
N/A
video
OTAvatar Ma et al. 2023
N/A
video
MakelfTalk Zhhou et al. 2020
X
X
audio
PC-AVS Zhou et al. 202
X
X
audio
RAD-NeRF (Tang et al.
2022
X
audio
Real3D-Portrai (ours)
V
audio/video
A
COMPARSION BETWEEN DIFFERENT METHODS
Our method also holds the idea of 3D reconstruction and 3D animation (Li al. 2023a) and et al. 2023b) do, but further improves the 3D reconstruction power by proposing an image-to -l-aplane (I2P) pretraining stage and enhances the face animation quality with motion adapter (MA), a PNCC- conditioned diff-plane predictor, hence achieves the goal of accurate 3D reconstruction and good animation quality.
The difference between our method and previous methods is obvious: Instead of previous end-to-end training, we propose a pretrain-anddfimttuu framework that simultaneously achieves the goals of accurate 3D reconstruction and stable face animation. To be specific,
RertrD- first distills 3D prior knowledge from a 3D GAN to tre-train an image-to-plane (I2P) model, then fine-tune the I2P model alongside motion adapter (MA) on video dataset to learn dynamic 3D talking face renderer, Besides, we are the first to consider natural torso movement and switchable background; Finally,
the first work that achieves both of audio and video-driven applications. For better comparison, we list the property of our method and several state-of-the-e baselines in Table
Our difference from HiDe-NeRE and GOS-Avatar As HiDe-NeRF (Li al. 2023a) and GOS- Avatar (Li et al.) 2023b) are the most relevant baselines of Parttrao that belong to the predictor-based one-shot 3D talking face generation paradigm,
discuss difference from them as follows. (1) We propose to pre-train the I2P model on an image multi-view dataset while the baselines don't; (2) Most importantly, we propose motion adapter that predicts motion diff-plane to directly morph the canonical tri-plane from the source expression into the target expression. By contrast, HiDe-NeRF learns deformation field, which results in bad geometry and bad visual quality (please refer to the video demo at Ec/static/viides
qithuu hatps//oompppiiiiaaee with deformationn mp4 for better demonstration);
and GOS-Avatar depends on extra expression neutralization to the source image, which requires additional supervision signals and may cause information loss in the source image. (3) We propose head-torso- torso--aarkooounddd paradigm that individually models the haad//toss/baccgroune segments, hence achieving realistic torso movement and overall good video naturalness. By contrast, the baselines don't 't consider the background and model the head-torso generic aadio-to-motion model to support the audio-driven task, while the baselines only support the
whole. (4) We propose video-driven task, which limits their usage in wealdworld applications.
B ADDITIONAL NETWORK AND TRAINING DETAILS
In this appendix, we present the detailed network structure and training details of ReataD-Portrait.
B. PRETRAINING MAGE-TO-PLANE MODEL
We perform a multi-view reconstruction (Trvvithick al.) 2023) task to effectively learn feed- forward I2P model that maps the input image to a 3D tri tri-plane representation. Specifically, we adopt a pre-trained EG3D generator to yield tri-planes P of various synthesized persons from the latent space.
14

---

Reference imiage 512K51ZK3
Reconstructed tripllane
Reference camera
camera
Image Plane Model
Volume Renderer
Tripplane
2-1(0.1)
FG3D Tri-plane
E630 Volume
Dual
Target
vGöfaan
camera
Predicted Image 512H512k3
Fixed Port
512N512K3
Learnble Part
Figure The pretraining process of 12P model in Sec.
Volume rendered image 128x x28x3
Super-resolution image 512x512x3
density
Volume Rendering
sigmoid
256x256x3
Conv2D Leaky
Conv2D Leaky RelU
(Linear ReLU) N
(Linear ReLU)
Upsample 2X
Upsample 2X
points
Coarse
rays
Sample
sampling
Sampler
Volume rendered image 128x128x3
triplane
camera
(a) Volume Renderer
(b) Super-Resolution Module
Figure 5: The network structure of volume renderer and naive super-resolution model used in the pretraining stage.
With the volume rendering technique, we could render images of the same person corresponding to the tri-plane P from arbitrary viewpoint given the camera parameters c. During training, to prepare the training data pair, we use an EG3D generator to generate tri ri-planes then use P to condition the volume renderer synthesize two images (Trf and Inv) of this identity from a reference camera Cer and a multi-view camera Cmv- Note that EG3D requires the camera to be sampled from cycle with fixed radius of 2.7, which is not desirable talking face generation, where we want an appropriate portion between the head and torso. Hence, we relax the fixed-radius constraint of the camera so the camera distance can range from 2.4 5.0. As shown in Fig our image-to-plan to-anane model takes the reference image Lef input reconstruct canonical tri-planc of the target person, then we volume renders the reconstructed tri-plane from the viewpoint of Cmv to render a multi-view image Imv- Intuitively, we use the error between Imv and Imv to provide a supervision signal to the image-to-plane model. Specifically, the training objective follows:
various identities,
L = MSE( Imv) + VGG (mm,, Imv) + VGG (nvv Imv) + DualAdví (Amv Imv)
where MSE is the mean-squared-erros between Im and mmv: VGG and VGG is the perceptual loss empowered by a pre-trained VGG19 (Simonyan Zissserman 2014) and VGGFace (Parkhi et al. 2015) network; DualAdy denotes the dual discrimination proposed by EG3D (Chan et al.) 2022). which could improve image fidelity and encourage keeping view consistency between the volume- rendered low-resolution image Iny and the super-solution image Iaw We used the pre-trained Dual Discriminator provided by Chan et al. (2022) and finetuned it during the training process.
B.2 OBTAINING PNCC FROM 3DMM COEIFICIENTS
We use the projected normalized coordinate code (PNCC) (Zhu et al. 2016 Li et al. 2023a) as the motion representation, which is appearance-agnoste feature image that only related the facial geometry, as the input condition to morph the 3D face. To be specific, PNCC can be formulated as:
PNCC Z-Buffer Vertee e), NCC), s.l. (ertex e) Vertex + Biii Besp
(
15

---

Preditted Image S1EHSLEK3
Background
larao Feature map
250025500
Deformation Capoecin adtes
Up Canvid Block
Bockground Festure map
Appertance Extractor
2562
25x
<256632
Head-Torun-Bo BB Occlusion aword Faaann
beform
Deform baned Decoder
Aoperrance Catnetoo
lcypoints
25xx3
Deform Featuren
C0 Luake NelU
N
Down Conv2d
Frilo
Heackaass-o
Appearance Cetractor
Upuample 2X
Keyponttite HestMMp
Derse Metton
ResBlock
K Nearest
harghber
inpartiing
Apprarance
lorto Branch
Extractor
Source
becogrtiund
Source
512H512K3
512x512x3
Prodcted hoad 12te12liug
target
keypoints
$12x512x3
(icad-Torso-Bll Brrss-Backkround
(b) Torso Brunch
(c) Background Branch
Super-Rect Resolution model
image
Reforence
Rendered head 1200c1200c2
Predicted image $12N11 2ock
Dual Adv, lass
512H5L2x3
Image Plane di
Parsing
Target image 512N512xd
Target
Volume Renderer
Head Torso Background Super Resolution model
512x512K(3
TH512H512K3
Estimator
$12+512xd
512x51224
Fixed Port
Parsing
Learnable Part
Figure 7: The training process of HTB-SR model in Sec. 3.3
where NCC is the normalized coordinate code provided by Zhu al. 2016) and acts the colormap in the Z-Buffer (Phong) 1998) rendering process; Vertex3p the vertex of reconstructed 3DMM face that in the canonical space, which is determined by 80 dimennnnno identity code and 64-dimension expression code e; Vertex Bia, and Beep are the template shape, identity basis, and expression basis of 3DMM model (Blanz & Vetter 1999). This way, could obtain PNCC from 3DMM identity/expression coefficients. Note that we could extract identity and expression coefficients from an image via 3DMM fitting. In the vidco-driven applications, we use the identity code from the source image and the expression code sequence from the driving video to construct the driving PNCC; as for the audio-driven applications, the required expression code sequence is predicted by the generic audio-to-motion model given the input audio.
B.3 WARPING-BASED TORSO BRANCH
In observation, it uncommon for the torso part rotate, and its dynamic be regarded as nearly a joint translation together with the head part. This observation motivates us to model the torso part with 2D warping-based renderer, which is computationally efficient and proven robust in complicated scenes (Siarohin et al.) 2019). As shown in Fig. b),), our torso branch can be viewed as a warping-basse based Face-vid2vid (Wang et al., (021] model that only renders the torso segment and is driven by predefined key points (instead of unsupervised keypoints (Wang et al. 2021) predicted from the image). be specific, firstly, we obtain the predefined keypoint with:
KP IdxKP(R. Vertexap
16

---

where Vertex3D is the 3DMM vertex reconstructed from identity and expression code defined in Eq. R and t is the rotation matrix and translation of the extracted camera, IdxKP denotes select 68 facial keypoints (Bulat & Tzimiropoulos 2017) from the 3DMM vertex. The key points are fed into the deformation motion estimator (DME) shown in Fig. (bb) to predict the deformation pixel coordinates of the torso segment. Then, we grid sample from the source torso appearance feature map with the
predicted deformation field to obtain the warped torso feature map. The overall warping-basee based torso rendering can be expressed as:
F Porso OBD(TAE((Fos))) DME(KPssc KPigt)
(10)
where TAE, DME, and DBD are the torso appearance encoder, dense motion estimator, and deformation- based decoder in Fig. 6 a). One could refer to (Wang et al. (02I)) for more details about these modules.
B.4 BACKGROUND BRANCH WITH KNN-BASED INPAINTING
The biggest challenge achieving realistic background is generating the pixels occupied by the foreground (i.e., the person) in the source image. To this end, we first adopt a K-nearest-neiihhor- based inpainting method to preprocess the background segment of the source image. Specifically, for each foreground pixel, we find its nearest neighbor that belongs to the background segment, then fill the foreground pixels with the color of their nearest background pixels. Once we obtained an inpainted background image, as shown in Fig. c)., we fed it into VGG-style appearance extractor to extract the background feature map. Note that since we individually model the background, we support switching backgrounds during inference. With the previous volume rendering, we obtain the low-resolution head image; with the torso branch and background branch, we obtain the torso and background feature map. Then, and generate a 512x512 composite image, which is shown in Fig. a).
super-reso resouuieso branch to integrate these three segments
B.5 OBTAINING THE HEAD AND TORSO OCCLUSION MASK
In Eq. 5 we propose to obtain the final talking portrait image with an alpha-blending-style fusion of head/torso/background feature maps. In this section, we introduce how to obtain the occlusion mask Maead and Mooso required in Eq. by utilizing the nature of NeRF-based head and warping-baed torso rendering module.
As for the NeRF- baseed head segment, following AD-NeRF (Guo et al. 2021). we could obtain head mask Maead by volume rendering:
Mhead 1." a(((t)) exp (- (-fr ((((s))ss) dt,
(11)
where a is the density predicted by the NeRF, and are the ray and ray marching depth of the volume rendering technique. and fy denote the nearest and farthest point of the ray.
As for the warping-based torso segment, following Face-vid2vid (Wang et al. 2021), when designing the dense motion estimator shown in Figure 6 a). apart from predicting the dense motion flow, the network also predicts I I-dimension occlusion mask for the torso Morso
B.6 DETAILED STRUCTURE OF AUDIO-TO-MOTION MODEL
We illustrate the detailed network structure of the A2M model in Fig The overall model consists of two generative models: a VAE as the main structure and Flow-based model as the enhanced prior of VAE. We use WavNet as the backbone of the encoder, decoder, and flow-based prior. 3DMM expression code, a 64-dimension vector, is chosen as the motion representation to be predicted, so the in-out dimension of the A2M model is T 64, where T is the time dimension. During inference. is convenient to obtain PNCC via the Z-Buffer algorithm given the predicted 3DMM expression code, illustrated in Appendix B.2
17

---

Input audio
H
HuBERT
Pitch
Landmarks
PNCC
Eye Blink
Mouth
Extra Factors
Amplitude
Input condition
Flow-based Prior
Expression
WavNet Encodier
WavNet Decoder
Expression Code
3DMM
A2M model
Figure 8: The network structure of A2M model. Dotted lines denote the processes that are only exccuted during the training phase in Sec. 3.4
C DETAILED MODEL CONFIGURATION
C.I MODEL CONFIGURATION
We provide detailed hyper-parameter settings about the model configuration in Table 6
Table 6: Model Configuration
Hyper-paramin
Value
ViT Branch Patch Size
VIT Branch Patch Emb Channel
1024
VIT Branch Attention Blocks
6
VIT Branch MLP Layers per Block
12P Model (87M)
VIT Branch Attention Block Channel
1024
VGG Branch Conv2D Layers
10
VGG Branch Conv2D Channels
256
Final Conv2D Layers
Output Feature Map Size
256 x 256 x 32
Patch Size
Patch Emb Channels
[32, 64, 160, 256] LayerNorm
Norm Type
Motion Adapter (5.7M)
Self-Attenton Block
MLP Layers per Block
Attention Heads
[1, 2, 5, 8]
Drop Path Rate
0.1
Discrimnator Dropout Rate
0.25
MLP Layers
Volume Renderer (0.004M)
MLP Channels
64
Torso Branch TAE -oovVDDDDDD Layers
3 3
Torso Branch MFE Conv3D Layers
Torso Branch DBD Conv2D Layers
BG Branch KNN number of neightbors
HTB-SR Model (50M)
BG Branch Appearance Encoder Conv2D layers
Head-Torso-BG Fusing Conv2D Layers
Conv2D/3D Kernel
Encoder WavNet Layers
8
Decoder WavNet Layers
Encoder/Dscoder Conv Kerel
A2M Model (10M)
Encoder/Decoder ConvID Channel Size
192
Latent Size
16
Prior Flow Layers
4
Prior Flow Conv ID Kernel
Prior Flow Conv ID Channel Size
18

---

C.2 TRAINING DETAILS.
All training processes of Real3tD-Pottritt performed 8 NVIDIA A100 GPUs. As for the renderer, we first pre-train the I2P model for 250,000 steps, which takes about 72 hours; then we train the motion adapter for 200,000 steps, which takes about 60 hours; finally, we train the HTB-SR model for 200,000 steps, which takes about 30 hours. As for the A2M model, we train it for 100,000 steps, which takes about 16 hours.
D ADDITIONAL EXPERIMENTS
D.1 EVALUATION DETAILS
In this section, we illustrate details for collecting the data for evaluation.
As for collecting the source image and driving audio/video, there are three groups of evaluated data: (1) for the Same-Identity Reenactment, we randomly chose 100 videos in our preserved validation split of CelebV-HOQ (2) for the Cross-Identity Reenactment, we use the first frame of the previously selected videos to obtain 100 identities, then use the expression-pose sequence from a randomly selected video construct the cross-identity reenactment data pair. (3) for the audio-driven scenario, we use 10 out-of-domain images downloaded from the internet (which are exactly the 10 identities shown in the demo video) and choose 10 audios from different languages to form the data pair (so each method has 100 vidcos as the test samples).
As for choosing the camera pose, since the prediction of the head pose is not main interest, we devise a naive strategy to obtain the head pose sequence from GT videos of V-H. To be specific, (1) in the same-identity setting, the driving head pose is exactly the same as the GT video of the source image (source image is the first frame of the test video); (2) in the cross-identity setting, the driving head pose is extracted from the driving video; (3) then, in the audio-driven setting, most importantly, we first estimate the head pose in the source image, then we query 10 videos in the CelebV- V-HQ that has nearest distance between its first frame's head pose and the source image. Once the candidate videos that provide head pose are sampled, we randomly choose one of them to drive the source image.
D.2 USER STUDY SETTING
We selected ten audio/video clips and ten identities to construct 100 talking portrait video samples for each audiovvideo-driven method. We involved 20 participants in rating each video. We perform the MOS evaluations from the aspect of identity preservation, visual quality, and audio-lip synchro- nization. Each tester is asked to evaluate the subjective score of video on a 1-5 Likert scale. For identity preservation, the source image and the video"; for visual quality, we tell the participants to "focus on the overall visual quality, including the image fidelity and smooth transition between adjacent frames": for audio-lip lip synchronization, we tell the participants to "only focus on the semantic- -eevel audio-lip synchronization, and ignores the visual quality".
tell the participants to "only focus on the similarity between the identity in
D.3 ADDITIONAL QUALITATIVE RESULTS
Overall Comparison. in Fig. and Fig 10
We provide a qualitative comparison with all video/audio-driver basclines
PNCC-conditioned Face Animation. We illustrate how PNCC animates the 3D avatar in Fig. 11 The first column the input head image of the image-to-plane model, and the second column is the input driving PNCC of the motion adapter model. The third column is the low-resolution rendered image (128 x 128) produced via volume rendering, and the fourth column is the corresponding depth image, which helps visualize the 3D geometry of the modeled 3D avatar. The fifth column shows the high-resolution rendered image (512 x 512) processed by the naive SR module used when training the motion adapter.
19

---

Pose 1
Pose
GT Image
Face Vid2Vid
OTAvatar
HiDe-NeRF
Ours
Figure 9: Qualitative Comparison with video-driven baselines. We recommend the reader refer to the demo video httpa://real 3dportrait qithub. 1c /ttsss/tiavvvoossssss// Compariscys wesoo_wtt__t__________ Leo__aaeeinnee mp4 for clear comparison. In this figure, we can see that (I) Face-vid2vid degrades at large head pose; (2) OTAvatar cannot produce
Senntty-pressevvngg result; (3) HiDe-NeRF produces texture jittering artifacts given different poses; and (4) our method could produce identity-preseee preeeeving and realistic results.
GT Image
MakeitTalk
PC-AVS
RAD-NeRF
Ours
Figure 10: Qualitative Comparison with audio-driven baselines. We recommend the reader refer to the demo video Comparisso ason with AD baselines mp4 for clear comparison. that (1) MakelrTalk tends produce cartoon-style results; (2) PC-AVS only renders the cropped face part and fails to achieve a good identity similarity; (3) our method and RAD-NeRF could generate identity-preservve and realistic talking portraits.
github. 1 /ts//tt/tivviooossss..../o
this figure, we can see
Realistic Torso Movement. As shown in Fig. 12 with the warping- based torso branch in the HTB-SR model, our method could generate realistic torso segments given large and critical head poses.
Switchable Background. shown in Fig. 13 with the background branch in the HTB-SR model, our method supports switching background during inference.
Audio-Lip Synchronization. As shown in Fig. 14 with the generic audio-to-motion model, our method achieves audio-lip synchronization the audio-driven scenario.
D.4 ADDITIONAL ABLATION STUDIES
Pretraining I2P Model on a Multi-view Image Dataset As shown in Fig. 15
without the
pretraining stage and learning the I2P model from scratch the video dataset leads to degraded
20

---

volume Rendered Image
volume Rendered Depth
Rendered Image after SR
Source Image
Driving PNCC
Figure 11: Illustration of how PNCC animates the 3D head.
Rendered Image (Pose
Rendered Image (Pose 2)
Rendered Image (Pose
Rendered Image (Pose 2)
Figure 12: Demonstration that our method could generate realistic torso segments given different head poses.
image quality and inaccurate facial animation. By contrast, with the pre-trained I2P model, we achieve high image fidelity, good identity-preservii ability, and accurate face motion control.
Fusion in HTB-SR Model We compare results with/without the proposed
head-tome sird-torso-bbcce barkk-aackgo alpha-blending fusion in Fig. 16 (1) As can be seen the first image, without alpha-blending, i.e., directly concatenating the feature maps of head/torso/backkrnn
results in hollow artifact in the hair region. We suspect that is caused by the unrestricted spatial information changing between these three semantic feature maps. For instance, in this case, the features from the background image override the information from the head image. By contrast, by using the face mask, we could eliminate the background information within the head region, which addresses the hollow artifact (as shown in the second image in Fig. 16). (2) Besides, as seen in the third image, without mask restrain the spatial feature fusion, the head-torso boundary region (marked by
21

---

E LIMITATIONS AND FUTURE WORK Source BG
Switched BG
Switched
Switched
Figure 13: Demonstration that our method supports switchable background.
past
up
that
don't
Figure Demonstration that our generic mudio-to-motion model could generate accurate lip motion.
Target Image
pretraining (Before SR)
w/o pretraining (After
with pretraining [Before
with pretraining [After 5R)
Figure 15: Comparison between with without the pretraining process of I2P model introduced in Sec. 3.1
red rectangle) seems unrealistic and blurry. By contrast, with the proposed alpha- blenddb fusion technique, we could generate realistic and sharp results in the boundary region.
D.5 ADDITIONAL QUANTITATIVE COMPARISON WITH RECENT 2D BASELINES
We additionally compare with several remarkable 2D baselines, such as DaGAN (Hong al. 2022a), TPS (Zhao & Zhang. 2022). DPE (Pang al. 2023). LIA (Wang al. 2022). and MCNet (Hong & Xu 2023), and provide a demo video for qualitative com- parison in https: /rwalldportraat..ttmmmm..aaaaa.t qithub.
with 1achh___adhh 5_additi ona 1_ vD Trp_basell leaaseline ines mp4 We also provide quantitative comparison in We can see that our method achieves the best performance in terms of CSIM, FID, AED and
Table 7
APD.
Firstly, due to the absence of harge-posed images in the training data, our method fails to generate images under large head poses like side views. We plan to address this problem introducing
22

---

w./o. alpha- blending
alpha- blending
w./o. alpha- blending
alpha- blending
Figure 16: Comparison between with or without alpha- blending-style head-torso- forso-background fusion introduced in Eq. [ We the rectangle point the artifacts caused by not using the alpha-bl hipha-blending fusion and use the rectangle with dotted to show that using alpha-blending fusion could address these problems.
Table 7: Additional comparison with 2D VD baselines.
Methods
CSIM
FID
AED
APD4
TPS (Zhao Zhang 2022)
0.745
43.54
0.137
0.028
DPE (Pang et al. 2023
0.731
44.07
0.135
0.021
MCNet (Hong & Xu (0233
0.758
42.49
0.136
0.030
PeattrD
0.764
41.58
0.129
0.017
more large-posed data and improving the tri-plane 3D representation. Secondly, the image quality can be improved by introducing more high-fidelity training data and more delicately designed networks. Thirdly, a few-shot in-context-lcarming 3D talking face method is desirable for better identity preservation and visual quality. Finally, though we have achieved generally high-quality realistic talking portrait results, one the limitations is that the inpainted background image could leak unnaturalness when the talking person is doing large pose motion. We believe the introduced naive KNN-based background inpainting method is to be blamed. Since generating a realistic background is of high importance to ensure the realism of the final video, plan upgrade the background inpainting method with a more advanced neural network-basee system, such as LAMA (Suvorov et al. 202T). An alternative might be learning the background inpainting module within the HTB-SR model in an end-to-end manner. We leave this for future work.
23


Reviews:
[{"review_id": "SDiKur8Vb1", "reviewer": "Reviewer_yTCz", "summary": "This work describes a method for single-shot 3D reconstruction and animation of a given 2D facial image. It uses an image to plane method to lift a 2D face into the tri-plane of a 3D GAN. It then morphs the triplane by adding a delta tri-plane which encoded the target expression via an PNCC encode image to animate the face. Differently from prior work the proposed method provides a solution for both video and audio driven animation of the face and also provides a super-resolution module for correctly morphing the torso and in-filling the exposed background regions. The proposed method is compared to several existing baseline video and audio-driven methods and shown to be superior them.", "questions": "I would like to see the authors' response to the questions I have raised in the weaknesses section of the paper above.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper addresses a fairly novel problem of single-shot joint 3D reconstruction and animation of facial images. Differently from prior work it seeks to inject the animation information directly into the 3D representation versus the dominant approach of animating in 2D first and then lifting into 3D. As the authors correctly point out animation in 3D versus 2D results is more correct handling of large head poses and less warping artifacts. So this is an important problem to address towards enabling large head pose facial talking head generation. \n\nIn comparison to the existing works on the topic, this work introduces a joint framework for both video and audio driven animation of 3D facial representations by encoding the audio signal into a PNCC representation. It also does a nice job of proposing to handle the torso and the background as a part of the overall solution thus enabling a more production-ready complete end-to-end framework. This is an often neglected detail in many works that treat the head in isolation from the backgrounds in which it exists requiring additional pre and post processing steps to deal with the torso and background.", "weaknesses": "1. The method by design requires the canonicalization, i.e., removal of the source image's facial expression, for the driving expression to be successfully applied to it. This is because the PNCC code is derived purely from the target driving video/audio's expression and hence cannot contain the information to erase/neutralize the source image's facial expression. I think the proposed method achieves this canonicalization during the fine-tuning phase with the Celeb-V-HQ video dataset. However, in my experience without any further stronger constraints, the proposed method cannot fully remove the source image's original facial expression. Can the author show examples of these canonicalized 3D face reconstructions of the input images without applying the target expression? Related to this is the question of how robust is the proposed method to the presence of large facial expressions in the source image. Does it work for expressions where the source image has a wide open mouth and lowered jaw or nearly closed eyes, for example?\n\n2. Overall the proposed methods is an obvious combination of several existing ideas from prior works on the topic to culminate in a successful large engineered end-to-end solution. For example, the idea of using I2P was previously proposed in LP3D; the idea of using rendered target expression images from a 3DMM was proposed in Li et al., 2023b, the audio to PNCC code predictor is borrowed from prior work; and the torso warping module is also borrowed from Wang et al., 2021. While the proposed combination of existing ideas results in an effective solution, from the research perspective the overall solution is light on significant novel or surprising insights.\n\n3. The authors don't specify the dataset/protocol that they used for evaluation.\n\n4. Comparisons of the proposed method to several of the newer (than Face-Vid2Vid) and better performing video-driven 2D facial animation methods are missing. Below I list several of them.\n\n[1] Thin-plate spline motion model for image animation., CVPR 2022\n\n[2] Depth-Aware Generative Adversarial Network for Talking Head Video Generation., CVPR 2022\n\n[3] FNeVR: Neural volume rendering for face animation., NeurIPS 2022\n\n[4] Latent Image Animator: Learning to Animate Images via Latent Space Navigation., ICLR 2022\n\n[5] DPE: Disentanglement of Pose and Expression for General Video Portrait Editing., CVPR 2023\n\n[6] Conditioned Memory Compensation Network for Talking Head video Generation., ICCV 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oa65jZE72W", "reviewer": "Reviewer_8dWt", "summary": "This paper proposes a method to create a reanimatable avatar from a single image. Leveraging the latent space of EG3D, the method learns to generate a canonical tri-plane from a given input image. This canonical triplane is then deformed using motion module conditioned on projected normalized co-ordinate code (PNCC). Finally, the deformed triplane is volume rendered to generate the final image. The motion conditioned tri-plane representation generates good results and outperforms some prior art, but the overall architecture and design is very similar to HiDe-NeRF.", "questions": "It would be great if the authors could clarify why the proposed method works better than HiDe-NeRF despite having a very similar architecture and set-up. Is pretraining the only reason?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1) The paper is well written.\n\n2) The qualitative results on reanimation are good, even when driven by audio.\n\n3) The quantitative results show that the proposed method out-performs prior art.\n\n4) The background is rendered well and merges seamlessly with the foreground.", "weaknesses": "1) Given that the overall architecture is very similar to HiDe-NeRF, it is unclear where the improvement of the proposed method is coming from. Is it because a pretrained Tri-plane is a better representation than the multi-resolution tri-plane features of HiDe-NeRF? It would be great if the authors could clarify this\n\n2) The addiction of background and torso modelling, while important, is relatively incremental.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VFoReYRHe5", "reviewer": "Reviewer_MF4S", "summary": "This paper proposes a novel method for synthesizing photorealistic talking portraits from a single source image coupled with either a speech signal or a driving facial video. The method generates a 3D avatar from the source image and then animates it based on the speech signal or the driving video. Apart from the talking head synthesis, the method pays attention in the realistic synthesis of torso movements and backgrounds. Regarding the module of 3D face reconstruction, the method adopts an image to plane model. Regarding the module of animation, the method introduces a novel facial motion adapter. The paper presents detailed experiments (qualitative and quantitative evaluations, user studies and ablation studies) that show the advantages and promising performance of the proposed method.", "questions": "- In the methodology, the paper does not clarify how the per-frame head pose is predicted from the audio, in the case of audio-driven talking face generation. More details and clarifications about that should be provided. \n\n- In terms of the realism of the final synthetic video, one can observe that the method has some limitations regarding generating realistic background. Considering the general quality and realism of the results, this is an acceptable limitation, but the authors should openly discuss these limitations and link them with potential directions of future works.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The proposed method is interesting and its pipeline has sufficient novelty, especially in terms of the combination of the large-scale image-to-plane backbone, the motion adapter and the Head-Torso-Background Super-Resolution model, which results in particularly realistic results. \n\n+ The paper includes an in-depth experimental evaluation that provides sound evidence about the promising results of the proposed method. In more detail, the proposed method is compared with several recent SOTA methods (despite the fact that some additional methods should have been included - see comments below). The evaluation includes qualitative and quantitative comparisons, as well as user studies that are important to judge the perceived quality of the results. The ablation studies are also detailed and clearly show the importance and benefits of the different modules of the pipeline. Finally, the supplementary videos are informative and help to appreciate the visual quality of the results, as well as the advantages of the proposed method over the previous SOTA techniques.", "weaknesses": "- The paper has omitted citing some important related methods of the field: \n\nJ. S. Chung, A. Jamaludin, and A. Zisserman, “You said that?” in BMVC, 2017.\n\nYe, Z., Xia, M., Yi, R., Zhang, J., Lai, Y.-K., Huang, X., et al. (2022). Audio-driven talking face video generation with dynamic convolution kernels. IEEE Transactions on Multimedia.\n\nIn addition, the method is based on the projected normalized coordinate code (PNCC) representation but it has not cited one of the most important works of the field that is also based on the same representation:\n\nKim, H., Garrido, P., Tewari, A., Xu, W., Thies, J., Niessner, M., Pérez, P., Richardt, C., Zollhöfer, M. and Theobalt, C., 2018. Deep video portraits. ACM transactions on graphics (TOG), 37(4), pp.1-14.\n\nAlso, the method supports the conditioning of the animation based on a speech audio signal but it does not cite another important and seminal work of the field that does something similar: \n\nKim, H., Elgharib, M., Zollhöfer, M., Seidel, H.P., Beeler, T., Richardt, C. and Theobalt, C., 2019. Neural style-preserving visual dubbing. ACM Transactions on Graphics (TOG), 38(6), pp.1-13.\n\n\n- In the part of the Audio-driven talking face generation of the experimental comparisons, some additional recent works should have been added in the comparisons. For example, the paper should have cited and included in the comparisons the following methods that solve the same problem:\n\nYi, R., Ye, Z., Zhang, J., Bao, H. and Liu, Y.J., 2020. Audio-driven talking face video generation with learning-based personalized head pose. arXiv preprint arXiv:2002.10137.\n\nYao, S., Zhong, R., Yan, Y., Zhai, G. and Yang, X., 2022. DFA-NeRF: Personalized talking head generation via disentangled face attributes neural rendering. arXiv preprint arXiv:2201.00791.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "brHsG9n1Xf", "reviewer": "Reviewer_Kntd", "summary": "The paper proposes a method for realistic one-shot 3D talking face generation driven by videos and audio. Given an input source image, the method first reconstructs a canonical 3D face in the tri-plane representation with an Image-to-Plane (I2P) model. The I2P comprises 1. a ViT branch with a stack of SegFormer blocks to handle the canonicalization and a VGG branch to capture the high-frequency appearance features. The driving motion is represented by the projected normalized coordinate code (PNCC) [Zhu et al. 2016]. Instead of using the deformation field to animate as in previous papers, the method proposes to have a motion adapter composed of a shallow SegFormer to output the residual motion diff-plane given the PNCC. The sum of the canonical tri-plane and the motion diff-plane are volume-rendered and upsampled by a super-resolution module to end up with the source image driven by the given motion. Lastly, the background and the torso are composited with the Head-Torso-Background supersampling model. The driving PNCC can come from audio using an audio-to-motion model using a flow-enhanced VAE or video by fitting 3DMM to the reference video. The method is compared against Face-vid2vid [Wang et al. 2021], OT-Avatar [Ma et al. 2023], HiDe-NeRF [Li et al. 2023a] for the video-driven animation, and MakeItTalk [Zhou et al. 2020], PC-AVS [Zhou et al. 2021], and RAD-NeRF [Tang et al. 2022] for the audio-driven animation.", "questions": "Can authors clarify which 3DMM they use? Is it BFM2009 as referred from section 3.4, or is it [Blanz and Vetter 1999] as referred from B.2? Note that while they both use PCA, their bases are not compatible because they are computed from different data.\n\n(Related to my question above)\nIn 3.4, authors say:\n> 3DMM basis is a nonhomogeneous linear equation and has multiple solutions (e.g., BFM2009)...\n\nAt least for PCA-based 3DMMs like BFM2009 and [Blanz and Vetter 1999], this is incorrect. Because PCA bases are orthogonal, a unique least squares solution can be found by simply multiplying the principal components and the data (target 3D mesh vertex positions in this case). Because of this misunderstanding, the paper may have an over-complex setup in its audio-to-motion model.\n\nHow does the I2P model handle an input image with a non-neutral face? I.e., if the model is given an image with an extreme facial expression as the source image, what happens? Have authors thought about canonicalization in this respect? I believe a concurrent work, \"Generalizable One-shot Neural Head Avatar,\" canonicalizes the input expression as well. Perhaps this should be clarified as a limitation.\n\nCan authors actually show results if the deformation field is used instead of the residual motion diff-plane? The paper just says \"bad quality,\" but this is never properly compared. The comparison video says the main issue with HiDe-NeRF (which uses the deformation field) is the temporal jittering. However, the temporal jittering is mitigated in this paper by the temporal Laplacian loss, not by the residual motion diff-plane.\n\nCan authors provide results from multiple views to show how good the 3D reconstruction is?\n\nCan authors consider comparisons to Next3D [Sun et al. 2023]? I understand that Next3D is an animatable 3D GAN model requiring a GAN inversion to reproduce an identity. A discussion on a concurrent work, \"Generalizable One-shot Neural Head Avatar,\" is not required but much appreciated to understand the novelty of this work.\n\n(minor typo)\n3.1 Network Design\n> comprises a stack of SegFormer bock\n\nbock -> block", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Out of the contributions that the authors claim, the motion adapter generating the residual motion diff-plane to animate the canonical tri-plane given the target PNCC seems to be the most interesting part of this work.", "weaknesses": "Some design decisions may not be well-supported. See my questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)